Diff of the two buildlogs: -- --- b1/build.log 2024-07-05 13:31:29.707269133 +0000 +++ b2/build.log 2024-07-05 13:37:52.110986264 +0000 @@ -1,6 +1,6 @@ I: pbuilder: network access will be disabled during build -I: Current time: Fri Jul 5 01:26:36 -12 2024 -I: pbuilder-time-stamp: 1720185996 +I: Current time: Sat Jul 6 03:31:39 +14 2024 +I: pbuilder-time-stamp: 1720186299 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/trixie-reproducible-base.tgz] I: copying local configuration @@ -28,52 +28,84 @@ dpkg-source: info: applying 0001-Skip-unreliable-tests.patch I: Not using root during the build. I: Installing the build-deps -I: user script /srv/workspace/pbuilder/18672/tmp/hooks/D02_print_environment starting +I: user script /srv/workspace/pbuilder/22201/tmp/hooks/D01_modify_environment starting +debug: Running on ff4a. +I: Changing host+domainname to test build reproducibility +I: Adding a custom variable just for the fun of it... +I: Changing /bin/sh to bash +'/bin/sh' -> '/bin/bash' +lrwxrwxrwx 1 root root 9 Jul 5 13:32 /bin/sh -> /bin/bash +I: Setting pbuilder2's login shell to /bin/bash +I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other +I: user script /srv/workspace/pbuilder/22201/tmp/hooks/D01_modify_environment finished +I: user script /srv/workspace/pbuilder/22201/tmp/hooks/D02_print_environment starting I: set - BUILDDIR='/build/reproducible-path' - BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' - BUILDUSERNAME='pbuilder1' - BUILD_ARCH='armhf' - DEBIAN_FRONTEND='noninteractive' - DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=3 ' - DISTRIBUTION='trixie' - HOME='/root' - HOST_ARCH='armhf' + BASH=/bin/sh + BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:globskipdots:hostcomplete:interactive_comments:patsub_replacement:progcomp:promptvars:sourcepath + BASH_ALIASES=() + BASH_ARGC=() + BASH_ARGV=() + BASH_CMDS=() + BASH_LINENO=([0]="12" [1]="0") + BASH_LOADABLES_PATH=/usr/local/lib/bash:/usr/lib/bash:/opt/local/lib/bash:/usr/pkg/lib/bash:/opt/pkg/lib/bash:. + BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment") + BASH_VERSINFO=([0]="5" [1]="2" [2]="21" [3]="1" [4]="release" [5]="arm-unknown-linux-gnueabihf") + BASH_VERSION='5.2.21(1)-release' + BUILDDIR=/build/reproducible-path + BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other' + BUILDUSERNAME=pbuilder2 + BUILD_ARCH=armhf + DEBIAN_FRONTEND=noninteractive + DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=4 ' + DIRSTACK=() + DISTRIBUTION=trixie + EUID=0 + FUNCNAME=([0]="Echo" [1]="main") + GROUPS=() + HOME=/root + HOSTNAME=i-capture-the-hostname + HOSTTYPE=arm + HOST_ARCH=armhf IFS=' ' - INVOCATION_ID='6e48224651024111929c6d05d6856be2' - LANG='C' - LANGUAGE='en_US:en' - LC_ALL='C' - MAIL='/var/mail/root' - OPTIND='1' - PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' - PBCURRENTCOMMANDLINEOPERATION='build' - PBUILDER_OPERATION='build' - PBUILDER_PKGDATADIR='/usr/share/pbuilder' - PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' - PBUILDER_SYSCONFDIR='/etc' - PPID='18672' - PS1='# ' - PS2='> ' + INVOCATION_ID=e17e1d8e6e724dbe85f99054661c3b25 + LANG=C + LANGUAGE=it_CH:it + LC_ALL=C + MACHTYPE=arm-unknown-linux-gnueabihf + MAIL=/var/mail/root + OPTERR=1 + OPTIND=1 + OSTYPE=linux-gnueabihf + PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path + PBCURRENTCOMMANDLINEOPERATION=build + PBUILDER_OPERATION=build + PBUILDER_PKGDATADIR=/usr/share/pbuilder + PBUILDER_PKGLIBDIR=/usr/lib/pbuilder + PBUILDER_SYSCONFDIR=/etc + PIPESTATUS=([0]="0") + POSIXLY_CORRECT=y + PPID=22201 PS4='+ ' - PWD='/' - SHELL='/bin/bash' - SHLVL='2' - SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.rB4Gqq2t/pbuilderrc_9QtQ --distribution trixie --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/trixie-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.rB4Gqq2t/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc' - SUDO_GID='110' - SUDO_UID='103' - SUDO_USER='jenkins' - TERM='unknown' - TZ='/usr/share/zoneinfo/Etc/GMT+12' - USER='root' - _='/usr/bin/systemd-run' - http_proxy='http://10.0.0.15:3142/' + PWD=/ + SHELL=/bin/bash + SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix + SHLVL=3 + SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.rB4Gqq2t/pbuilderrc_SjW8 --distribution trixie --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/trixie-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.rB4Gqq2t/b2 --logfile b2/build.log golang-github-hashicorp-raft_1.5.0-1.dsc' + SUDO_GID=113 + SUDO_UID=107 + SUDO_USER=jenkins + TERM=unknown + TZ=/usr/share/zoneinfo/Etc/GMT-14 + UID=0 + USER=root + _='I: set' + http_proxy=http://10.0.0.15:3142/ I: uname -a - Linux virt64z 6.1.0-22-arm64 #1 SMP Debian 6.1.94-1 (2024-06-21) aarch64 GNU/Linux + Linux i-capture-the-hostname 6.1.0-22-armmp-lpae #1 SMP Debian 6.1.94-1 (2024-06-21) armv7l GNU/Linux I: ls -l /bin lrwxrwxrwx 1 root root 7 Jun 5 22:35 /bin -> usr/bin -I: user script /srv/workspace/pbuilder/18672/tmp/hooks/D02_print_environment finished +I: user script /srv/workspace/pbuilder/22201/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy @@ -223,7 +255,7 @@ Get: 100 http://deb.debian.org/debian trixie/main armhf golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 101 http://deb.debian.org/debian trixie/main armhf golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 102 http://deb.debian.org/debian trixie/main armhf golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] -Fetched 86.4 MB in 3s (33.9 MB/s) +Fetched 86.4 MB in 6s (15.3 MB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package sensible-utils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19412 files and directories currently installed.) @@ -646,7 +678,11 @@ Building tag database... -> Finished parsing the build-deps I: Building the package -I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.5.0-1_source.changes +I: user script /srv/workspace/pbuilder/22201/tmp/hooks/A99_set_merged_usr starting +Not re-configuring usrmerge for trixie +I: user script /srv/workspace/pbuilder/22201/tmp/hooks/A99_set_merged_usr finished +hostname: Name or service not known +I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.5.0-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.5.0-1 dpkg-buildpackage: info: source distribution unstable @@ -664,20 +700,20 @@ dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang - cd obj-arm-linux-gnueabihf && go install -trimpath -v -p 3 github.com/hashicorp/raft github.com/hashicorp/raft/bench -internal/goarch + cd obj-arm-linux-gnueabihf && go install -trimpath -v -p 4 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/unsafeheader -internal/cpu +internal/goarch +internal/godebugs internal/coverage/rtcov +internal/cpu internal/abi internal/chacha8rand -internal/bytealg -internal/godebugs internal/goexperiment internal/goos -runtime/internal/atomic runtime/internal/math runtime/internal/sys +internal/bytealg +runtime/internal/atomic runtime/internal/syscall internal/race sync/atomic @@ -685,140 +721,140 @@ unicode/utf8 container/list internal/itoa -runtime math/bits -math crypto/internal/alias crypto/internal/boring/sig +math cmp slices encoding unicode/utf16 log/internal internal/nettrace +runtime vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/alias +crypto/subtle internal/reflectlite sync -crypto/subtle -internal/testlog internal/bisect internal/singleflight +internal/testlog runtime/cgo -internal/godebug -sort errors +sort io -bytes -strings +internal/godebug internal/oserror syscall -bufio +bytes +strings hash strconv +bufio crypto/internal/randutil internal/safefilepath path -crypto math/rand +crypto reflect +github.com/hashicorp/golang-lru/simplelru time internal/syscall/unix internal/syscall/execenv -github.com/hashicorp/golang-lru/simplelru github.com/hashicorp/go-immutable-radix vendor/golang.org/x/net/dns/dnsmessage +internal/intern +net/netip context internal/poll io/fs -internal/intern -net/netip +hash/crc32 +crypto/internal/nistec/fiat +crypto/rc4 +vendor/golang.org/x/text/transform +net/http/internal/ascii +regexp/syntax os encoding/binary -internal/fmtsort -hash/crc32 crypto/cipher -crypto/internal/nistec/fiat -crypto/internal/boring -fmt +internal/fmtsort encoding/base64 +fmt +crypto/internal/boring +crypto/aes net +crypto/des +crypto/internal/edwards25519/field +crypto/sha512 +crypto/internal/edwards25519 math/big encoding/json -crypto/rand log compress/flate -crypto/aes compress/gzip -crypto/des -crypto/internal/edwards25519/field crypto/internal/nistec +crypto/hmac +crypto/md5 +crypto/sha1 +crypto/rand crypto/internal/bigmod crypto/internal/boring/bbig -crypto/sha512 encoding/asn1 crypto/ecdh crypto/elliptic -vendor/golang.org/x/crypto/cryptobyte -crypto/internal/edwards25519 -crypto/ecdsa crypto/ed25519 -crypto/hmac -crypto/md5 -crypto/rc4 crypto/rsa -crypto/sha1 crypto/sha256 crypto/dsa +vendor/golang.org/x/crypto/cryptobyte encoding/hex encoding/pem crypto/x509/pkix net/url path/filepath +crypto/ecdsa vendor/golang.org/x/crypto/chacha20 vendor/golang.org/x/crypto/internal/poly1305 vendor/golang.org/x/crypto/hkdf vendor/golang.org/x/crypto/chacha20poly1305 -vendor/golang.org/x/text/transform vendor/golang.org/x/text/unicode/bidi vendor/golang.org/x/text/unicode/norm -crypto/x509 -vendor/golang.org/x/text/secure/bidirule -net/textproto vendor/golang.org/x/net/http2/hpack -vendor/golang.org/x/net/idna mime +vendor/golang.org/x/text/secure/bidirule mime/quotedprintable -crypto/tls -mime/multipart -vendor/golang.org/x/net/http/httpguts -vendor/golang.org/x/net/http/httpproxy net/http/internal -net/http/internal/ascii os/signal golang.org/x/sys/unix io/ioutil -regexp/syntax regexp +vendor/golang.org/x/net/idna internal/saferio encoding/gob -net/http/httptrace -net/http -github.com/mattn/go-isatty go/token -github.com/mattn/go-colorable -github.com/fatih/color html -github.com/hashicorp/go-hclog +crypto/x509 +net/textproto +vendor/golang.org/x/net/http/httpguts +vendor/golang.org/x/net/http/httpproxy +mime/multipart text/template/parse hash/crc64 flag +github.com/mattn/go-isatty internal/sysinfo +github.com/mattn/go-colorable runtime/debug +crypto/tls +github.com/fatih/color text/template runtime/trace +github.com/hashicorp/go-hclog testing html/template +net/http/httptrace +net/http net/rpc github.com/armon/go-metrics github.com/hashicorp/go-msgpack/codec @@ -827,7 +863,7 @@ debian/rules override_dh_auto_test make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' dh_auto_test - cd obj-arm-linux-gnueabihf && go test -vet=off -v -p 3 github.com/hashicorp/raft github.com/hashicorp/raft/bench + cd obj-arm-linux-gnueabihf && go test -vet=off -v -p 4 github.com/hashicorp/raft github.com/hashicorp/raft/bench ? github.com/hashicorp/raft/bench [no test files] === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) @@ -870,32 +906,32 @@ === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir -2024-07-05T13:29:19.236Z [INFO] creating new snapshot: path=/tmp/raft2021643987/raft644205749/snapshots/3-10-1720186159236.tmp +2024-07-05T13:35:34.011Z [INFO] creating new snapshot: path=/tmp/raft2825948441/raft1792033498/snapshots/3-10-1720186534011.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.01s) === RUN TestFileSS_CreateSnapshot -2024-07-05T13:29:19.243Z [INFO] creating new snapshot: path=/tmp/raft2459832826/snapshots/3-10-1720186159243.tmp -2024-07-05T13:29:19.248Z [WARN] found temporary snapshot: name=3-10-1720186159243.tmp +2024-07-05T13:35:34.024Z [INFO] creating new snapshot: path=/tmp/raft1775188664/snapshots/3-10-1720186534024.tmp +2024-07-05T13:35:34.030Z [WARN] found temporary snapshot: name=3-10-1720186534024.tmp --- PASS: TestFileSS_CreateSnapshot (0.02s) === RUN TestFileSS_CancelSnapshot -2024-07-05T13:29:19.264Z [INFO] creating new snapshot: path=/tmp/raft15462951/snapshots/3-10-1720186159264.tmp +2024-07-05T13:35:34.047Z [INFO] creating new snapshot: path=/tmp/raft1590669862/snapshots/3-10-1720186534047.tmp --- PASS: TestFileSS_CancelSnapshot (0.01s) === RUN TestFileSS_Retention -2024-07-05T13:29:19.273Z [INFO] creating new snapshot: path=/tmp/raft1264235282/snapshots/3-10-1720186159273.tmp -2024-07-05T13:29:19.288Z [INFO] creating new snapshot: path=/tmp/raft1264235282/snapshots/3-11-1720186159288.tmp -2024-07-05T13:29:19.304Z [INFO] creating new snapshot: path=/tmp/raft1264235282/snapshots/3-12-1720186159304.tmp -2024-07-05T13:29:19.319Z [INFO] reaping snapshot: path=/tmp/raft1264235282/snapshots/3-10-1720186159273 -2024-07-05T13:29:19.320Z [INFO] creating new snapshot: path=/tmp/raft1264235282/snapshots/3-13-1720186159319.tmp -2024-07-05T13:29:19.336Z [INFO] reaping snapshot: path=/tmp/raft1264235282/snapshots/3-11-1720186159288 -2024-07-05T13:29:19.337Z [INFO] creating new snapshot: path=/tmp/raft1264235282/snapshots/3-14-1720186159337.tmp -2024-07-05T13:29:19.362Z [INFO] reaping snapshot: path=/tmp/raft1264235282/snapshots/3-12-1720186159304 ---- PASS: TestFileSS_Retention (0.09s) +2024-07-05T13:35:34.060Z [INFO] creating new snapshot: path=/tmp/raft392547114/snapshots/3-10-1720186534060.tmp +2024-07-05T13:35:34.082Z [INFO] creating new snapshot: path=/tmp/raft392547114/snapshots/3-11-1720186534082.tmp +2024-07-05T13:35:34.102Z [INFO] creating new snapshot: path=/tmp/raft392547114/snapshots/3-12-1720186534102.tmp +2024-07-05T13:35:34.123Z [INFO] reaping snapshot: path=/tmp/raft392547114/snapshots/3-10-1720186534060 +2024-07-05T13:35:34.124Z [INFO] creating new snapshot: path=/tmp/raft392547114/snapshots/3-13-1720186534124.tmp +2024-07-05T13:35:34.144Z [INFO] reaping snapshot: path=/tmp/raft392547114/snapshots/3-11-1720186534082 +2024-07-05T13:35:34.144Z [INFO] creating new snapshot: path=/tmp/raft392547114/snapshots/3-14-1720186534144.tmp +2024-07-05T13:35:34.164Z [INFO] reaping snapshot: path=/tmp/raft392547114/snapshots/3-12-1720186534102 +--- PASS: TestFileSS_Retention (0.11s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering -2024-07-05T13:29:19.365Z [INFO] creating new snapshot: path=/tmp/raft2111623832/snapshots/5-130350-1720186159365.tmp -2024-07-05T13:29:19.384Z [INFO] creating new snapshot: path=/tmp/raft2111623832/snapshots/36-204917-1720186159384.tmp +2024-07-05T13:35:34.169Z [INFO] creating new snapshot: path=/tmp/raft3193298914/snapshots/5-130350-1720186534169.tmp +2024-07-05T13:35:34.190Z [INFO] creating new snapshot: path=/tmp/raft3193298914/snapshots/36-204917-1720186534190.tmp --- PASS: TestFileSS_Ordering (0.04s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) @@ -932,7 +968,7 @@ === RUN TestEmitsLogStoreMetrics --- PASS: TestEmitsLogStoreMetrics (0.01s) === RUN TestNetworkTransport_CloseStreams ---- PASS: TestNetworkTransport_CloseStreams (0.01s) +--- PASS: TestNetworkTransport_CloseStreams (0.02s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath @@ -942,24 +978,24 @@ === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.01s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams ---- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.03s) +--- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.04s) === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 +2024-07-05T13:35:34.366Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 -2024-07-05T13:29:19.526Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" +2024-07-05T13:35:34.424Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 -2024-07-05T13:29:19.578Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" ---- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.16s) +2024-07-05T13:35:34.481Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" +--- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.17s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s) - --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s) - --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s) + --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.06s) + --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.06s) === RUN TestNetworkTransport_RequestVote -2024-07-05T13:29:19.633Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" ---- PASS: TestNetworkTransport_RequestVote (0.00s) +--- PASS: TestNetworkTransport_RequestVote (0.01s) === RUN TestNetworkTransport_InstallSnapshot ---- PASS: TestNetworkTransport_InstallSnapshot (0.01s) +--- PASS: TestNetworkTransport_InstallSnapshot (0.02s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider @@ -967,27 +1003,27 @@ === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.01s) === RUN TestNetworkTransport_ListenBackoff - net_transport_test.go:839: countingWriter: 2024-07-05T13:29:19.654Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2024-07-05T13:35:34.518Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2024-07-05T13:29:19.659Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2024-07-05T13:35:34.524Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2024-07-05T13:29:19.670Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2024-07-05T13:35:34.534Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2024-07-05T13:29:19.692Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2024-07-05T13:35:34.555Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2024-07-05T13:29:19.732Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2024-07-05T13:35:34.596Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2024-07-05T13:29:19.813Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2024-07-05T13:35:34.677Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2024-07-05T13:29:19.976Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2024-07-05T13:35:34.837Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2024-07-05T13:29:20.296Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2024-07-05T13:35:35.159Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2024-07-05T13:29:20.936Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2024-07-05T13:35:35.800Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2024-07-05T13:29:21.940Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2024-07-05T13:35:36.802Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2024-07-05T13:29:22.943Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2024-07-05T13:35:37.805Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration @@ -997,2536 +1033,2556 @@ === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop -2024-07-05T13:29:23.658Z [INFO] server-bc9f0a36-699d-43ac-d1a9-dcd6ce5b743b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc9f0a36-699d-43ac-d1a9-dcd6ce5b743b Address:bc9f0a36-699d-43ac-d1a9-dcd6ce5b743b}]" +2024-07-05T13:35:38.530Z [INFO] server-511e04dc-4ebb-82aa-bfda-b27258f1ac83: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-511e04dc-4ebb-82aa-bfda-b27258f1ac83 Address:511e04dc-4ebb-82aa-bfda-b27258f1ac83}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown -2024-07-05T13:29:23.659Z [INFO] server-45251a9d-b856-eac4-85f9-680eae7fb67c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45251a9d-b856-eac4-85f9-680eae7fb67c Address:45251a9d-b856-eac4-85f9-680eae7fb67c}]" +2024-07-05T13:35:38.534Z [INFO] server-c386192f-809a-6632-7cd4-98fcceb5c79a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c386192f-809a-6632-7cd4-98fcceb5c79a Address:c386192f-809a-6632-7cd4-98fcceb5c79a}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap -2024-07-05T13:29:23.661Z [INFO] server-1b0b5cb0-852f-2b9a-d77d-370e309a1575: initial configuration: index=0 servers=[] -2024-07-05T13:29:23.661Z [INFO] server-cf93cc43-f46a-b07b-7020-0f2058b0ff02: initial configuration: index=0 servers=[] -2024-07-05T13:29:23.664Z [INFO] server-cf93cc43-f46a-b07b-7020-0f2058b0ff02: entering follower state: follower="Node at cf93cc43-f46a-b07b-7020-0f2058b0ff02 [Follower]" leader-address= leader-id= -2024-07-05T13:29:23.664Z [INFO] server-72e3d3a5-7830-3202-043b-263a96d5cb7e: initial configuration: index=0 servers=[] -2024-07-05T13:29:23.661Z [INFO] server-1b0b5cb0-852f-2b9a-d77d-370e309a1575: entering follower state: follower="Node at 1b0b5cb0-852f-2b9a-d77d-370e309a1575 [Follower]" leader-address= leader-id= -2024-07-05T13:29:23.664Z [INFO] server-72e3d3a5-7830-3202-043b-263a96d5cb7e: entering follower state: follower="Node at 72e3d3a5-7830-3202-043b-263a96d5cb7e [Follower]" leader-address= leader-id= -2024-07-05T13:29:23.664Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:29:23.720Z [WARN] server-1b0b5cb0-852f-2b9a-d77d-370e309a1575: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:23.720Z [INFO] server-1b0b5cb0-852f-2b9a-d77d-370e309a1575: entering candidate state: node="Node at 1b0b5cb0-852f-2b9a-d77d-370e309a1575 [Candidate]" term=2 -2024-07-05T13:29:23.720Z [INFO] server-1b0b5cb0-852f-2b9a-d77d-370e309a1575: election won: term=2 tally=2 -2024-07-05T13:29:23.720Z [INFO] server-1b0b5cb0-852f-2b9a-d77d-370e309a1575: entering leader state: leader="Node at 1b0b5cb0-852f-2b9a-d77d-370e309a1575 [Leader]" -2024-07-05T13:29:23.720Z [INFO] server-1b0b5cb0-852f-2b9a-d77d-370e309a1575: added peer, starting replication: peer=server-cf93cc43-f46a-b07b-7020-0f2058b0ff02 -2024-07-05T13:29:23.720Z [INFO] server-1b0b5cb0-852f-2b9a-d77d-370e309a1575: added peer, starting replication: peer=server-72e3d3a5-7830-3202-043b-263a96d5cb7e -2024-07-05T13:29:23.720Z [WARN] server-cf93cc43-f46a-b07b-7020-0f2058b0ff02: failed to get previous log: previous-index=1 last-index=0 error="log not found" -2024-07-05T13:29:23.721Z [WARN] server-72e3d3a5-7830-3202-043b-263a96d5cb7e: failed to get previous log: previous-index=1 last-index=0 error="log not found" -2024-07-05T13:29:23.721Z [WARN] server-1b0b5cb0-852f-2b9a-d77d-370e309a1575: appendEntries rejected, sending older logs: peer="{Voter server-cf93cc43-f46a-b07b-7020-0f2058b0ff02 cf93cc43-f46a-b07b-7020-0f2058b0ff02}" next=1 -2024-07-05T13:29:23.721Z [WARN] server-1b0b5cb0-852f-2b9a-d77d-370e309a1575: appendEntries rejected, sending older logs: peer="{Voter server-72e3d3a5-7830-3202-043b-263a96d5cb7e 72e3d3a5-7830-3202-043b-263a96d5cb7e}" next=1 -2024-07-05T13:29:23.721Z [INFO] server-1b0b5cb0-852f-2b9a-d77d-370e309a1575: pipelining replication: peer="{Voter server-cf93cc43-f46a-b07b-7020-0f2058b0ff02 cf93cc43-f46a-b07b-7020-0f2058b0ff02}" -2024-07-05T13:29:23.721Z [INFO] server-1b0b5cb0-852f-2b9a-d77d-370e309a1575: pipelining replication: peer="{Voter server-72e3d3a5-7830-3202-043b-263a96d5cb7e 72e3d3a5-7830-3202-043b-263a96d5cb7e}" -2024-07-05T13:29:23.826Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:29:23.720906867 +0000 UTC m=+4.490817542 (2 nodes), 56.253427ms from start of poll, 59.859501ms from cluster start. Timeout at 2024-07-05 13:29:23.826018499 +0000 UTC m=+4.595929174, 105.111632ms after stability -2024-07-05T13:29:23.826Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:23.932Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:23.826208853 +0000 UTC m=+4.596119528 (1 nodes), 1.48µs from start of poll, 165.161487ms from cluster start. Timeout at 2024-07-05 13:29:23.93261428 +0000 UTC m=+4.702524955, 106.405427ms after stability -2024-07-05T13:29:23.943Z [INFO] server-1b0b5cb0-852f-2b9a-d77d-370e309a1575: aborting pipeline replication: peer="{Voter server-72e3d3a5-7830-3202-043b-263a96d5cb7e 72e3d3a5-7830-3202-043b-263a96d5cb7e}" -2024-07-05T13:29:23.943Z [INFO] server-1b0b5cb0-852f-2b9a-d77d-370e309a1575: aborting pipeline replication: peer="{Voter server-cf93cc43-f46a-b07b-7020-0f2058b0ff02 cf93cc43-f46a-b07b-7020-0f2058b0ff02}" ---- PASS: TestRaft_LiveBootstrap (0.29s) +2024-07-05T13:35:38.541Z [INFO] server-af02f59e-9b75-df90-6fa3-c3574ddfed4d: initial configuration: index=0 servers=[] +2024-07-05T13:35:38.542Z [INFO] server-baec8ab2-58e6-8c80-a1b8-6d6c0381d024: initial configuration: index=0 servers=[] +2024-07-05T13:35:38.542Z [INFO] server-af02f59e-9b75-df90-6fa3-c3574ddfed4d: entering follower state: follower="Node at af02f59e-9b75-df90-6fa3-c3574ddfed4d [Follower]" leader-address= leader-id= +2024-07-05T13:35:38.542Z [INFO] server-93f192b4-0519-6459-ea1a-133038f16222: initial configuration: index=0 servers=[] +2024-07-05T13:35:38.542Z [INFO] server-baec8ab2-58e6-8c80-a1b8-6d6c0381d024: entering follower state: follower="Node at baec8ab2-58e6-8c80-a1b8-6d6c0381d024 [Follower]" leader-address= leader-id= +2024-07-05T13:35:38.543Z [INFO] server-93f192b4-0519-6459-ea1a-133038f16222: entering follower state: follower="Node at 93f192b4-0519-6459-ea1a-133038f16222 [Follower]" leader-address= leader-id= +2024-07-05T13:35:38.543Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:35:38.623Z [WARN] server-baec8ab2-58e6-8c80-a1b8-6d6c0381d024: no known peers, aborting election +2024-07-05T13:35:38.637Z [WARN] server-93f192b4-0519-6459-ea1a-133038f16222: no known peers, aborting election +2024-07-05T13:35:38.637Z [WARN] server-af02f59e-9b75-df90-6fa3-c3574ddfed4d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:38.637Z [INFO] server-af02f59e-9b75-df90-6fa3-c3574ddfed4d: entering candidate state: node="Node at af02f59e-9b75-df90-6fa3-c3574ddfed4d [Candidate]" term=2 +2024-07-05T13:35:38.638Z [INFO] server-af02f59e-9b75-df90-6fa3-c3574ddfed4d: election won: term=2 tally=2 +2024-07-05T13:35:38.638Z [INFO] server-af02f59e-9b75-df90-6fa3-c3574ddfed4d: entering leader state: leader="Node at af02f59e-9b75-df90-6fa3-c3574ddfed4d [Leader]" +2024-07-05T13:35:38.638Z [INFO] server-af02f59e-9b75-df90-6fa3-c3574ddfed4d: added peer, starting replication: peer=server-baec8ab2-58e6-8c80-a1b8-6d6c0381d024 +2024-07-05T13:35:38.638Z [INFO] server-af02f59e-9b75-df90-6fa3-c3574ddfed4d: added peer, starting replication: peer=server-93f192b4-0519-6459-ea1a-133038f16222 +2024-07-05T13:35:38.639Z [WARN] server-baec8ab2-58e6-8c80-a1b8-6d6c0381d024: failed to get previous log: previous-index=1 last-index=0 error="log not found" +2024-07-05T13:35:38.639Z [WARN] server-93f192b4-0519-6459-ea1a-133038f16222: failed to get previous log: previous-index=1 last-index=0 error="log not found" +2024-07-05T13:35:38.639Z [WARN] server-af02f59e-9b75-df90-6fa3-c3574ddfed4d: appendEntries rejected, sending older logs: peer="{Voter server-baec8ab2-58e6-8c80-a1b8-6d6c0381d024 baec8ab2-58e6-8c80-a1b8-6d6c0381d024}" next=1 +2024-07-05T13:35:38.639Z [WARN] server-af02f59e-9b75-df90-6fa3-c3574ddfed4d: appendEntries rejected, sending older logs: peer="{Voter server-93f192b4-0519-6459-ea1a-133038f16222 93f192b4-0519-6459-ea1a-133038f16222}" next=1 +2024-07-05T13:35:38.640Z [INFO] server-af02f59e-9b75-df90-6fa3-c3574ddfed4d: pipelining replication: peer="{Voter server-baec8ab2-58e6-8c80-a1b8-6d6c0381d024 baec8ab2-58e6-8c80-a1b8-6d6c0381d024}" +2024-07-05T13:35:38.640Z [INFO] server-af02f59e-9b75-df90-6fa3-c3574ddfed4d: pipelining replication: peer="{Voter server-93f192b4-0519-6459-ea1a-133038f16222 93f192b4-0519-6459-ea1a-133038f16222}" +2024-07-05T13:35:38.743Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:35:38.638514224 +0000 UTC m=+4.634435675 (2 nodes), 94.824059ms from start of poll, 96.810856ms from cluster start. Timeout at 2024-07-05 13:35:38.743543222 +0000 UTC m=+4.739464673, 105.028998ms after stability +2024-07-05T13:35:38.744Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:38.849Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:38.744146086 +0000 UTC m=+4.740067537 (1 nodes), 6.416µs from start of poll, 202.442718ms from cluster start. Timeout at 2024-07-05 13:35:38.849598284 +0000 UTC m=+4.845519736, 105.452199ms after stability +2024-07-05T13:35:38.862Z [INFO] server-af02f59e-9b75-df90-6fa3-c3574ddfed4d: aborting pipeline replication: peer="{Voter server-93f192b4-0519-6459-ea1a-133038f16222 93f192b4-0519-6459-ea1a-133038f16222}" +2024-07-05T13:35:38.862Z [INFO] server-af02f59e-9b75-df90-6fa3-c3574ddfed4d: aborting pipeline replication: peer="{Voter server-baec8ab2-58e6-8c80-a1b8-6d6c0381d024 baec8ab2-58e6-8c80-a1b8-6d6c0381d024}" +--- PASS: TestRaft_LiveBootstrap (0.33s) === RUN TestRaft_LiveBootstrap_From_NonVoter -2024-07-05T13:29:23.946Z [INFO] server-26fa0e79-95ea-e5ac-67b4-672b0113fdff: initial configuration: index=0 servers=[] -2024-07-05T13:29:23.946Z [INFO] server-75ad911b-9096-ab99-4bc7-4ed9a77c1645: initial configuration: index=0 servers=[] -2024-07-05T13:29:23.946Z [INFO] server-26fa0e79-95ea-e5ac-67b4-672b0113fdff: entering follower state: follower="Node at 26fa0e79-95ea-e5ac-67b4-672b0113fdff [Follower]" leader-address= leader-id= ---- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s) +2024-07-05T13:35:38.870Z [INFO] server-022b15ea-0e5d-06c2-f03f-93b7d7ce9ef4: initial configuration: index=0 servers=[] +2024-07-05T13:35:38.871Z [INFO] server-89d869c6-b2cf-0e82-3d71-4c0e16f433b6: initial configuration: index=0 servers=[] +2024-07-05T13:35:38.871Z [INFO] server-022b15ea-0e5d-06c2-f03f-93b7d7ce9ef4: entering follower state: follower="Node at 022b15ea-0e5d-06c2-f03f-93b7d7ce9ef4 [Follower]" leader-address= leader-id= +2024-07-05T13:35:38.871Z [INFO] server-89d869c6-b2cf-0e82-3d71-4c0e16f433b6: entering follower state: follower="Node at 89d869c6-b2cf-0e82-3d71-4c0e16f433b6 [Follower]" leader-address= leader-id= +--- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.01s) === RUN TestRaft_RecoverCluster_NoState -2024-07-05T13:29:23.947Z [INFO] server-5bb335cb-4f61-08da-6f3b-4fc1a192fe8a: initial configuration: index=0 servers=[] +2024-07-05T13:35:38.878Z [INFO] server-20aa882c-4614-c17c-3af9-f022b59388d5: initial configuration: index=0 servers=[] +2024-07-05T13:35:38.878Z [INFO] server-20aa882c-4614-c17c-3af9-f022b59388d5: entering follower state: follower="Node at 20aa882c-4614-c17c-3af9-f022b59388d5 [Shutdown]" leader-address= leader-id= --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster === RUN TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs -2024-07-05T13:29:23.949Z [INFO] server-8e86ff91-933c-7e4c-01fc-c7d9585cc996: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e86ff91-933c-7e4c-01fc-c7d9585cc996 Address:8e86ff91-933c-7e4c-01fc-c7d9585cc996} {Suffrage:Voter ID:server-9092716b-b492-e333-1dfa-f5b1815128a1 Address:9092716b-b492-e333-1dfa-f5b1815128a1} {Suffrage:Voter ID:server-928e9350-64b4-eb5b-8e33-2e753498e62e Address:928e9350-64b4-eb5b-8e33-2e753498e62e}]" -2024-07-05T13:29:23.949Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e86ff91-933c-7e4c-01fc-c7d9585cc996 Address:8e86ff91-933c-7e4c-01fc-c7d9585cc996} {Suffrage:Voter ID:server-9092716b-b492-e333-1dfa-f5b1815128a1 Address:9092716b-b492-e333-1dfa-f5b1815128a1} {Suffrage:Voter ID:server-928e9350-64b4-eb5b-8e33-2e753498e62e Address:928e9350-64b4-eb5b-8e33-2e753498e62e}]" -2024-07-05T13:29:23.949Z [INFO] server-928e9350-64b4-eb5b-8e33-2e753498e62e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e86ff91-933c-7e4c-01fc-c7d9585cc996 Address:8e86ff91-933c-7e4c-01fc-c7d9585cc996} {Suffrage:Voter ID:server-9092716b-b492-e333-1dfa-f5b1815128a1 Address:9092716b-b492-e333-1dfa-f5b1815128a1} {Suffrage:Voter ID:server-928e9350-64b4-eb5b-8e33-2e753498e62e Address:928e9350-64b4-eb5b-8e33-2e753498e62e}]" -2024-07-05T13:29:23.949Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:23.949Z [INFO] server-8e86ff91-933c-7e4c-01fc-c7d9585cc996: entering follower state: follower="Node at 8e86ff91-933c-7e4c-01fc-c7d9585cc996 [Follower]" leader-address= leader-id= -2024-07-05T13:29:23.949Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: entering follower state: follower="Node at 9092716b-b492-e333-1dfa-f5b1815128a1 [Follower]" leader-address= leader-id= -2024-07-05T13:29:23.949Z [INFO] server-928e9350-64b4-eb5b-8e33-2e753498e62e: entering follower state: follower="Node at 928e9350-64b4-eb5b-8e33-2e753498e62e [Follower]" leader-address= leader-id= -2024-07-05T13:29:24.016Z [WARN] server-9092716b-b492-e333-1dfa-f5b1815128a1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:24.016Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: entering candidate state: node="Node at 9092716b-b492-e333-1dfa-f5b1815128a1 [Candidate]" term=2 -2024-07-05T13:29:24.016Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: election won: term=2 tally=2 -2024-07-05T13:29:24.016Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: entering leader state: leader="Node at 9092716b-b492-e333-1dfa-f5b1815128a1 [Leader]" -2024-07-05T13:29:24.016Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: added peer, starting replication: peer=server-8e86ff91-933c-7e4c-01fc-c7d9585cc996 -2024-07-05T13:29:24.016Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: added peer, starting replication: peer=server-928e9350-64b4-eb5b-8e33-2e753498e62e -2024-07-05T13:29:24.017Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: pipelining replication: peer="{Voter server-928e9350-64b4-eb5b-8e33-2e753498e62e 928e9350-64b4-eb5b-8e33-2e753498e62e}" -2024-07-05T13:29:24.017Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: pipelining replication: peer="{Voter server-8e86ff91-933c-7e4c-01fc-c7d9585cc996 8e86ff91-933c-7e4c-01fc-c7d9585cc996}" -2024-07-05T13:29:24.128Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:24.017356667 +0000 UTC m=+4.787267342 (1 nodes), 67.540391ms from start of poll, 68.213887ms from cluster start. Timeout at 2024-07-05 13:29:24.128254537 +0000 UTC m=+4.898165212, 110.89787ms after stability -2024-07-05T13:29:24.128Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: aborting pipeline replication: peer="{Voter server-928e9350-64b4-eb5b-8e33-2e753498e62e 928e9350-64b4-eb5b-8e33-2e753498e62e}" -2024-07-05T13:29:24.128Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: aborting pipeline replication: peer="{Voter server-8e86ff91-933c-7e4c-01fc-c7d9585cc996 8e86ff91-933c-7e4c-01fc-c7d9585cc996}" -2024-07-05T13:29:24.128Z [INFO] creating new snapshot: path=/tmp/raft3668130337/snapshots/2-2-1720186164128.tmp -2024-07-05T13:29:24.130Z [INFO] server-8e86ff91-933c-7e4c-01fc-c7d9585cc996: starting restore from snapshot: id=2-2-1720186164128 last-index=2 last-term=2 size-in-bytes=1 -2024-07-05T13:29:24.130Z [INFO] server-8e86ff91-933c-7e4c-01fc-c7d9585cc996: snapshot restore progress: id=2-2-1720186164128 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% -2024-07-05T13:29:24.130Z [INFO] server-8e86ff91-933c-7e4c-01fc-c7d9585cc996: restored from snapshot: id=2-2-1720186164128 last-index=2 last-term=2 size-in-bytes=1 -2024-07-05T13:29:24.130Z [INFO] server-8e86ff91-933c-7e4c-01fc-c7d9585cc996: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e86ff91-933c-7e4c-01fc-c7d9585cc996 Address:8e86ff91-933c-7e4c-01fc-c7d9585cc996} {Suffrage:Voter ID:server-9092716b-b492-e333-1dfa-f5b1815128a1 Address:9092716b-b492-e333-1dfa-f5b1815128a1} {Suffrage:Voter ID:server-928e9350-64b4-eb5b-8e33-2e753498e62e Address:928e9350-64b4-eb5b-8e33-2e753498e62e}]" -2024-07-05T13:29:24.130Z [INFO] creating new snapshot: path=/tmp/raft3232412997/snapshots/2-2-1720186164130.tmp -2024-07-05T13:29:24.131Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: starting restore from snapshot: id=2-2-1720186164130 last-index=2 last-term=2 size-in-bytes=1 -2024-07-05T13:29:24.131Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: snapshot restore progress: id=2-2-1720186164130 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% -2024-07-05T13:29:24.131Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: restored from snapshot: id=2-2-1720186164130 last-index=2 last-term=2 size-in-bytes=1 -2024-07-05T13:29:24.131Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e86ff91-933c-7e4c-01fc-c7d9585cc996 Address:8e86ff91-933c-7e4c-01fc-c7d9585cc996} {Suffrage:Voter ID:server-9092716b-b492-e333-1dfa-f5b1815128a1 Address:9092716b-b492-e333-1dfa-f5b1815128a1} {Suffrage:Voter ID:server-928e9350-64b4-eb5b-8e33-2e753498e62e Address:928e9350-64b4-eb5b-8e33-2e753498e62e}]" -2024-07-05T13:29:24.132Z [INFO] creating new snapshot: path=/tmp/raft3955940917/snapshots/2-2-1720186164132.tmp -2024-07-05T13:29:24.136Z [INFO] server-8e86ff91-933c-7e4c-01fc-c7d9585cc996: entering follower state: follower="Node at 8e86ff91-933c-7e4c-01fc-c7d9585cc996 [Follower]" leader-address= leader-id= -2024-07-05T13:29:24.136Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: entering follower state: follower="Node at 9092716b-b492-e333-1dfa-f5b1815128a1 [Follower]" leader-address= leader-id= -2024-07-05T13:29:24.192Z [WARN] server-8e86ff91-933c-7e4c-01fc-c7d9585cc996: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:24.192Z [INFO] server-8e86ff91-933c-7e4c-01fc-c7d9585cc996: entering candidate state: node="Node at 8e86ff91-933c-7e4c-01fc-c7d9585cc996 [Candidate]" term=3 -2024-07-05T13:29:24.192Z [ERROR] server-8e86ff91-933c-7e4c-01fc-c7d9585cc996: failed to make requestVote RPC: target="{Voter server-928e9350-64b4-eb5b-8e33-2e753498e62e 928e9350-64b4-eb5b-8e33-2e753498e62e}" error="failed to connect to peer: 928e9350-64b4-eb5b-8e33-2e753498e62e" term=3 -2024-07-05T13:29:24.192Z [ERROR] server-8e86ff91-933c-7e4c-01fc-c7d9585cc996: failed to make requestVote RPC: target="{Voter server-9092716b-b492-e333-1dfa-f5b1815128a1 9092716b-b492-e333-1dfa-f5b1815128a1}" error="failed to connect to peer: 9092716b-b492-e333-1dfa-f5b1815128a1" term=3 -2024-07-05T13:29:24.193Z [INFO] server-928e9350-64b4-eb5b-8e33-2e753498e62e: starting restore from snapshot: id=2-2-1720186164132 last-index=2 last-term=2 size-in-bytes=1 -2024-07-05T13:29:24.194Z [INFO] server-928e9350-64b4-eb5b-8e33-2e753498e62e: snapshot restore progress: id=2-2-1720186164132 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% -2024-07-05T13:29:24.194Z [INFO] server-928e9350-64b4-eb5b-8e33-2e753498e62e: restored from snapshot: id=2-2-1720186164132 last-index=2 last-term=2 size-in-bytes=1 -2024-07-05T13:29:24.194Z [INFO] server-928e9350-64b4-eb5b-8e33-2e753498e62e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e86ff91-933c-7e4c-01fc-c7d9585cc996 Address:8e86ff91-933c-7e4c-01fc-c7d9585cc996} {Suffrage:Voter ID:server-9092716b-b492-e333-1dfa-f5b1815128a1 Address:9092716b-b492-e333-1dfa-f5b1815128a1} {Suffrage:Voter ID:server-928e9350-64b4-eb5b-8e33-2e753498e62e Address:928e9350-64b4-eb5b-8e33-2e753498e62e}]" -2024-07-05T13:29:24.194Z [INFO] server-928e9350-64b4-eb5b-8e33-2e753498e62e: entering follower state: follower="Node at 928e9350-64b4-eb5b-8e33-2e753498e62e [Follower]" leader-address= leader-id= -2024-07-05T13:29:24.235Z [WARN] server-9092716b-b492-e333-1dfa-f5b1815128a1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:24.235Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: entering candidate state: node="Node at 9092716b-b492-e333-1dfa-f5b1815128a1 [Candidate]" term=3 -2024-07-05T13:29:24.235Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: election won: term=3 tally=2 -2024-07-05T13:29:24.235Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: entering leader state: leader="Node at 9092716b-b492-e333-1dfa-f5b1815128a1 [Leader]" -2024-07-05T13:29:24.235Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: added peer, starting replication: peer=server-8e86ff91-933c-7e4c-01fc-c7d9585cc996 -2024-07-05T13:29:24.235Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: added peer, starting replication: peer=server-928e9350-64b4-eb5b-8e33-2e753498e62e -2024-07-05T13:29:24.235Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: pipelining replication: peer="{Voter server-928e9350-64b4-eb5b-8e33-2e753498e62e 928e9350-64b4-eb5b-8e33-2e753498e62e}" -2024-07-05T13:29:24.235Z [INFO] server-8e86ff91-933c-7e4c-01fc-c7d9585cc996: duplicate requestVote for same term: term=3 -2024-07-05T13:29:24.235Z [INFO] server-8e86ff91-933c-7e4c-01fc-c7d9585cc996: entering follower state: follower="Node at 8e86ff91-933c-7e4c-01fc-c7d9585cc996 [Follower]" leader-address=9092716b-b492-e333-1dfa-f5b1815128a1 leader-id=server-9092716b-b492-e333-1dfa-f5b1815128a1 -2024-07-05T13:29:24.235Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: pipelining replication: peer="{Voter server-8e86ff91-933c-7e4c-01fc-c7d9585cc996 8e86ff91-933c-7e4c-01fc-c7d9585cc996}" -2024-07-05T13:29:24.511Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:24.616Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:24.511415135 +0000 UTC m=+5.281325810 (1 nodes), 75.917µs from start of poll, 562.272355ms from cluster start. Timeout at 2024-07-05 13:29:24.616524408 +0000 UTC m=+5.386435083, 105.109273ms after stability -2024-07-05T13:29:24.616Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: aborting pipeline replication: peer="{Voter server-928e9350-64b4-eb5b-8e33-2e753498e62e 928e9350-64b4-eb5b-8e33-2e753498e62e}" -2024-07-05T13:29:24.616Z [INFO] server-9092716b-b492-e333-1dfa-f5b1815128a1: aborting pipeline replication: peer="{Voter server-8e86ff91-933c-7e4c-01fc-c7d9585cc996 8e86ff91-933c-7e4c-01fc-c7d9585cc996}" +2024-07-05T13:35:38.886Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e10818d-a168-0682-f9e6-7b048b9517af Address:6e10818d-a168-0682-f9e6-7b048b9517af} {Suffrage:Voter ID:server-c60852b9-3d91-abb3-e008-9f13ee660d06 Address:c60852b9-3d91-abb3-e008-9f13ee660d06} {Suffrage:Voter ID:server-629d3da6-cb17-e36d-2f3b-78841ca90c63 Address:629d3da6-cb17-e36d-2f3b-78841ca90c63}]" +2024-07-05T13:35:38.887Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: entering follower state: follower="Node at 6e10818d-a168-0682-f9e6-7b048b9517af [Follower]" leader-address= leader-id= +2024-07-05T13:35:38.887Z [INFO] server-c60852b9-3d91-abb3-e008-9f13ee660d06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e10818d-a168-0682-f9e6-7b048b9517af Address:6e10818d-a168-0682-f9e6-7b048b9517af} {Suffrage:Voter ID:server-c60852b9-3d91-abb3-e008-9f13ee660d06 Address:c60852b9-3d91-abb3-e008-9f13ee660d06} {Suffrage:Voter ID:server-629d3da6-cb17-e36d-2f3b-78841ca90c63 Address:629d3da6-cb17-e36d-2f3b-78841ca90c63}]" +2024-07-05T13:35:38.887Z [INFO] server-c60852b9-3d91-abb3-e008-9f13ee660d06: entering follower state: follower="Node at c60852b9-3d91-abb3-e008-9f13ee660d06 [Follower]" leader-address= leader-id= +2024-07-05T13:35:38.888Z [INFO] server-629d3da6-cb17-e36d-2f3b-78841ca90c63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e10818d-a168-0682-f9e6-7b048b9517af Address:6e10818d-a168-0682-f9e6-7b048b9517af} {Suffrage:Voter ID:server-c60852b9-3d91-abb3-e008-9f13ee660d06 Address:c60852b9-3d91-abb3-e008-9f13ee660d06} {Suffrage:Voter ID:server-629d3da6-cb17-e36d-2f3b-78841ca90c63 Address:629d3da6-cb17-e36d-2f3b-78841ca90c63}]" +2024-07-05T13:35:38.888Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:38.888Z [INFO] server-629d3da6-cb17-e36d-2f3b-78841ca90c63: entering follower state: follower="Node at 629d3da6-cb17-e36d-2f3b-78841ca90c63 [Follower]" leader-address= leader-id= +2024-07-05T13:35:38.974Z [WARN] server-6e10818d-a168-0682-f9e6-7b048b9517af: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:38.974Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: entering candidate state: node="Node at 6e10818d-a168-0682-f9e6-7b048b9517af [Candidate]" term=2 +2024-07-05T13:35:38.975Z [WARN] server-c60852b9-3d91-abb3-e008-9f13ee660d06: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:38.975Z [INFO] server-c60852b9-3d91-abb3-e008-9f13ee660d06: entering candidate state: node="Node at c60852b9-3d91-abb3-e008-9f13ee660d06 [Candidate]" term=2 +2024-07-05T13:35:38.975Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: election won: term=2 tally=2 +2024-07-05T13:35:38.975Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: entering leader state: leader="Node at 6e10818d-a168-0682-f9e6-7b048b9517af [Leader]" +2024-07-05T13:35:38.975Z [INFO] server-c60852b9-3d91-abb3-e008-9f13ee660d06: duplicate requestVote for same term: term=2 +2024-07-05T13:35:38.975Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: added peer, starting replication: peer=server-c60852b9-3d91-abb3-e008-9f13ee660d06 +2024-07-05T13:35:38.975Z [INFO] server-629d3da6-cb17-e36d-2f3b-78841ca90c63: duplicate requestVote for same term: term=2 +2024-07-05T13:35:38.975Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: added peer, starting replication: peer=server-629d3da6-cb17-e36d-2f3b-78841ca90c63 +2024-07-05T13:35:38.976Z [INFO] server-c60852b9-3d91-abb3-e008-9f13ee660d06: entering follower state: follower="Node at c60852b9-3d91-abb3-e008-9f13ee660d06 [Follower]" leader-address=6e10818d-a168-0682-f9e6-7b048b9517af leader-id=server-6e10818d-a168-0682-f9e6-7b048b9517af +2024-07-05T13:35:38.976Z [WARN] server-6e10818d-a168-0682-f9e6-7b048b9517af: rejecting vote request since we have a leader: from=c60852b9-3d91-abb3-e008-9f13ee660d06 leader=6e10818d-a168-0682-f9e6-7b048b9517af leader-id=server-6e10818d-a168-0682-f9e6-7b048b9517af +2024-07-05T13:35:38.976Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: pipelining replication: peer="{Voter server-c60852b9-3d91-abb3-e008-9f13ee660d06 c60852b9-3d91-abb3-e008-9f13ee660d06}" +2024-07-05T13:35:38.977Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: pipelining replication: peer="{Voter server-629d3da6-cb17-e36d-2f3b-78841ca90c63 629d3da6-cb17-e36d-2f3b-78841ca90c63}" +2024-07-05T13:35:39.082Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:38.976541841 +0000 UTC m=+4.972463292 (1 nodes), 87.611274ms from start of poll, 90.499305ms from cluster start. Timeout at 2024-07-05 13:35:39.082623153 +0000 UTC m=+5.078544896, 106.081604ms after stability +2024-07-05T13:35:39.083Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: aborting pipeline replication: peer="{Voter server-629d3da6-cb17-e36d-2f3b-78841ca90c63 629d3da6-cb17-e36d-2f3b-78841ca90c63}" +2024-07-05T13:35:39.084Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: aborting pipeline replication: peer="{Voter server-c60852b9-3d91-abb3-e008-9f13ee660d06 c60852b9-3d91-abb3-e008-9f13ee660d06}" +2024-07-05T13:35:39.084Z [INFO] creating new snapshot: path=/tmp/raft2197042506/snapshots/2-2-1720186539084.tmp +2024-07-05T13:35:39.101Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: starting restore from snapshot: id=2-2-1720186539084 last-index=2 last-term=2 size-in-bytes=1 +2024-07-05T13:35:39.102Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: snapshot restore progress: id=2-2-1720186539084 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% +2024-07-05T13:35:39.103Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: restored from snapshot: id=2-2-1720186539084 last-index=2 last-term=2 size-in-bytes=1 +2024-07-05T13:35:39.103Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e10818d-a168-0682-f9e6-7b048b9517af Address:6e10818d-a168-0682-f9e6-7b048b9517af} {Suffrage:Voter ID:server-c60852b9-3d91-abb3-e008-9f13ee660d06 Address:c60852b9-3d91-abb3-e008-9f13ee660d06} {Suffrage:Voter ID:server-629d3da6-cb17-e36d-2f3b-78841ca90c63 Address:629d3da6-cb17-e36d-2f3b-78841ca90c63}]" +2024-07-05T13:35:39.103Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: entering follower state: follower="Node at 6e10818d-a168-0682-f9e6-7b048b9517af [Follower]" leader-address= leader-id= +2024-07-05T13:35:39.104Z [INFO] creating new snapshot: path=/tmp/raft975717595/snapshots/2-2-1720186539103.tmp +2024-07-05T13:35:39.114Z [INFO] server-c60852b9-3d91-abb3-e008-9f13ee660d06: starting restore from snapshot: id=2-2-1720186539103 last-index=2 last-term=2 size-in-bytes=1 +2024-07-05T13:35:39.116Z [INFO] server-c60852b9-3d91-abb3-e008-9f13ee660d06: snapshot restore progress: id=2-2-1720186539103 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% +2024-07-05T13:35:39.116Z [INFO] server-c60852b9-3d91-abb3-e008-9f13ee660d06: restored from snapshot: id=2-2-1720186539103 last-index=2 last-term=2 size-in-bytes=1 +2024-07-05T13:35:39.116Z [INFO] server-c60852b9-3d91-abb3-e008-9f13ee660d06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e10818d-a168-0682-f9e6-7b048b9517af Address:6e10818d-a168-0682-f9e6-7b048b9517af} {Suffrage:Voter ID:server-c60852b9-3d91-abb3-e008-9f13ee660d06 Address:c60852b9-3d91-abb3-e008-9f13ee660d06} {Suffrage:Voter ID:server-629d3da6-cb17-e36d-2f3b-78841ca90c63 Address:629d3da6-cb17-e36d-2f3b-78841ca90c63}]" +2024-07-05T13:35:39.117Z [INFO] server-c60852b9-3d91-abb3-e008-9f13ee660d06: entering follower state: follower="Node at c60852b9-3d91-abb3-e008-9f13ee660d06 [Follower]" leader-address= leader-id= +2024-07-05T13:35:39.117Z [INFO] creating new snapshot: path=/tmp/raft3052790634/snapshots/2-2-1720186539117.tmp +2024-07-05T13:35:39.131Z [INFO] server-629d3da6-cb17-e36d-2f3b-78841ca90c63: starting restore from snapshot: id=2-2-1720186539117 last-index=2 last-term=2 size-in-bytes=1 +2024-07-05T13:35:39.132Z [INFO] server-629d3da6-cb17-e36d-2f3b-78841ca90c63: snapshot restore progress: id=2-2-1720186539117 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% +2024-07-05T13:35:39.133Z [INFO] server-629d3da6-cb17-e36d-2f3b-78841ca90c63: restored from snapshot: id=2-2-1720186539117 last-index=2 last-term=2 size-in-bytes=1 +2024-07-05T13:35:39.133Z [INFO] server-629d3da6-cb17-e36d-2f3b-78841ca90c63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e10818d-a168-0682-f9e6-7b048b9517af Address:6e10818d-a168-0682-f9e6-7b048b9517af} {Suffrage:Voter ID:server-c60852b9-3d91-abb3-e008-9f13ee660d06 Address:c60852b9-3d91-abb3-e008-9f13ee660d06} {Suffrage:Voter ID:server-629d3da6-cb17-e36d-2f3b-78841ca90c63 Address:629d3da6-cb17-e36d-2f3b-78841ca90c63}]" +2024-07-05T13:35:39.133Z [INFO] server-629d3da6-cb17-e36d-2f3b-78841ca90c63: entering follower state: follower="Node at 629d3da6-cb17-e36d-2f3b-78841ca90c63 [Follower]" leader-address= leader-id= +2024-07-05T13:35:39.176Z [WARN] server-6e10818d-a168-0682-f9e6-7b048b9517af: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:39.176Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: entering candidate state: node="Node at 6e10818d-a168-0682-f9e6-7b048b9517af [Candidate]" term=3 +2024-07-05T13:35:39.177Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: election won: term=3 tally=2 +2024-07-05T13:35:39.177Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: entering leader state: leader="Node at 6e10818d-a168-0682-f9e6-7b048b9517af [Leader]" +2024-07-05T13:35:39.177Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: added peer, starting replication: peer=server-c60852b9-3d91-abb3-e008-9f13ee660d06 +2024-07-05T13:35:39.178Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: added peer, starting replication: peer=server-629d3da6-cb17-e36d-2f3b-78841ca90c63 +2024-07-05T13:35:39.178Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: pipelining replication: peer="{Voter server-c60852b9-3d91-abb3-e008-9f13ee660d06 c60852b9-3d91-abb3-e008-9f13ee660d06}" +2024-07-05T13:35:39.179Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: pipelining replication: peer="{Voter server-629d3da6-cb17-e36d-2f3b-78841ca90c63 629d3da6-cb17-e36d-2f3b-78841ca90c63}" +2024-07-05T13:35:39.448Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:39.555Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:39.449597866 +0000 UTC m=+5.445519318 (1 nodes), 349.702µs from start of poll, 563.555331ms from cluster start. Timeout at 2024-07-05 13:35:39.55501769 +0000 UTC m=+5.550939142, 105.419824ms after stability +2024-07-05T13:35:39.555Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: aborting pipeline replication: peer="{Voter server-629d3da6-cb17-e36d-2f3b-78841ca90c63 629d3da6-cb17-e36d-2f3b-78841ca90c63}" +2024-07-05T13:35:39.555Z [INFO] server-6e10818d-a168-0682-f9e6-7b048b9517af: aborting pipeline replication: peer="{Voter server-c60852b9-3d91-abb3-e008-9f13ee660d06 c60852b9-3d91-abb3-e008-9f13ee660d06}" === RUN TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs -2024-07-05T13:29:24.622Z [INFO] server-d6145d04-3871-a651-e7f2-397dee798ad9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6145d04-3871-a651-e7f2-397dee798ad9 Address:d6145d04-3871-a651-e7f2-397dee798ad9} {Suffrage:Voter ID:server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb Address:f65b7af9-fb9a-ed59-d7fa-56e7391d43bb} {Suffrage:Voter ID:server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d Address:d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d}]" -2024-07-05T13:29:24.622Z [INFO] server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6145d04-3871-a651-e7f2-397dee798ad9 Address:d6145d04-3871-a651-e7f2-397dee798ad9} {Suffrage:Voter ID:server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb Address:f65b7af9-fb9a-ed59-d7fa-56e7391d43bb} {Suffrage:Voter ID:server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d Address:d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d}]" -2024-07-05T13:29:24.622Z [INFO] server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6145d04-3871-a651-e7f2-397dee798ad9 Address:d6145d04-3871-a651-e7f2-397dee798ad9} {Suffrage:Voter ID:server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb Address:f65b7af9-fb9a-ed59-d7fa-56e7391d43bb} {Suffrage:Voter ID:server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d Address:d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d}]" -2024-07-05T13:29:24.622Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:24.622Z [INFO] server-d6145d04-3871-a651-e7f2-397dee798ad9: entering follower state: follower="Node at d6145d04-3871-a651-e7f2-397dee798ad9 [Follower]" leader-address= leader-id= -2024-07-05T13:29:24.623Z [INFO] server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb: entering follower state: follower="Node at f65b7af9-fb9a-ed59-d7fa-56e7391d43bb [Follower]" leader-address= leader-id= -2024-07-05T13:29:24.623Z [INFO] server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d: entering follower state: follower="Node at d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d [Follower]" leader-address= leader-id= -2024-07-05T13:29:24.675Z [WARN] server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:24.675Z [INFO] server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d: entering candidate state: node="Node at d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d [Candidate]" term=2 -2024-07-05T13:29:24.675Z [INFO] server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d: election won: term=2 tally=2 -2024-07-05T13:29:24.675Z [INFO] server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d: entering leader state: leader="Node at d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d [Leader]" -2024-07-05T13:29:24.675Z [INFO] server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d: added peer, starting replication: peer=server-d6145d04-3871-a651-e7f2-397dee798ad9 -2024-07-05T13:29:24.675Z [INFO] server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d: added peer, starting replication: peer=server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb -2024-07-05T13:29:24.675Z [INFO] server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d: pipelining replication: peer="{Voter server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb f65b7af9-fb9a-ed59-d7fa-56e7391d43bb}" -2024-07-05T13:29:24.675Z [INFO] server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d: pipelining replication: peer="{Voter server-d6145d04-3871-a651-e7f2-397dee798ad9 d6145d04-3871-a651-e7f2-397dee798ad9}" -2024-07-05T13:29:24.781Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:24.67603396 +0000 UTC m=+5.445944635 (1 nodes), 53.120576ms from start of poll, 56.041754ms from cluster start. Timeout at 2024-07-05 13:29:24.781881487 +0000 UTC m=+5.551792162, 105.847527ms after stability -2024-07-05T13:29:24.783Z [INFO] server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d: aborting pipeline replication: peer="{Voter server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb f65b7af9-fb9a-ed59-d7fa-56e7391d43bb}" -2024-07-05T13:29:24.783Z [INFO] server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d: aborting pipeline replication: peer="{Voter server-d6145d04-3871-a651-e7f2-397dee798ad9 d6145d04-3871-a651-e7f2-397dee798ad9}" -2024-07-05T13:29:24.783Z [INFO] creating new snapshot: path=/tmp/raft1384285123/snapshots/2-6-1720186164783.tmp -2024-07-05T13:29:24.785Z [INFO] server-d6145d04-3871-a651-e7f2-397dee798ad9: starting restore from snapshot: id=2-6-1720186164783 last-index=6 last-term=2 size-in-bytes=25 -2024-07-05T13:29:24.785Z [INFO] server-d6145d04-3871-a651-e7f2-397dee798ad9: snapshot restore progress: id=2-6-1720186164783 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% -2024-07-05T13:29:24.785Z [INFO] server-d6145d04-3871-a651-e7f2-397dee798ad9: restored from snapshot: id=2-6-1720186164783 last-index=6 last-term=2 size-in-bytes=25 -2024-07-05T13:29:24.785Z [INFO] server-d6145d04-3871-a651-e7f2-397dee798ad9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6145d04-3871-a651-e7f2-397dee798ad9 Address:d6145d04-3871-a651-e7f2-397dee798ad9} {Suffrage:Voter ID:server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb Address:f65b7af9-fb9a-ed59-d7fa-56e7391d43bb} {Suffrage:Voter ID:server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d Address:d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d}]" -2024-07-05T13:29:24.786Z [INFO] creating new snapshot: path=/tmp/raft2864374372/snapshots/2-6-1720186164786.tmp -2024-07-05T13:29:24.787Z [INFO] server-d6145d04-3871-a651-e7f2-397dee798ad9: entering follower state: follower="Node at d6145d04-3871-a651-e7f2-397dee798ad9 [Follower]" leader-address= leader-id= -2024-07-05T13:29:24.787Z [INFO] server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb: starting restore from snapshot: id=2-6-1720186164786 last-index=6 last-term=2 size-in-bytes=25 -2024-07-05T13:29:24.788Z [INFO] server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb: snapshot restore progress: id=2-6-1720186164786 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% -2024-07-05T13:29:24.788Z [INFO] server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb: restored from snapshot: id=2-6-1720186164786 last-index=6 last-term=2 size-in-bytes=25 -2024-07-05T13:29:24.788Z [INFO] server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6145d04-3871-a651-e7f2-397dee798ad9 Address:d6145d04-3871-a651-e7f2-397dee798ad9} {Suffrage:Voter ID:server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb Address:f65b7af9-fb9a-ed59-d7fa-56e7391d43bb} {Suffrage:Voter ID:server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d Address:d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d}]" -2024-07-05T13:29:24.788Z [INFO] creating new snapshot: path=/tmp/raft3455653632/snapshots/2-6-1720186164788.tmp -2024-07-05T13:29:24.789Z [INFO] server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb: entering follower state: follower="Node at f65b7af9-fb9a-ed59-d7fa-56e7391d43bb [Follower]" leader-address= leader-id= -2024-07-05T13:29:24.789Z [INFO] server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d: starting restore from snapshot: id=2-6-1720186164788 last-index=6 last-term=2 size-in-bytes=25 -2024-07-05T13:29:24.790Z [INFO] server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d: snapshot restore progress: id=2-6-1720186164788 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% -2024-07-05T13:29:24.790Z [INFO] server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d: restored from snapshot: id=2-6-1720186164788 last-index=6 last-term=2 size-in-bytes=25 -2024-07-05T13:29:24.790Z [INFO] server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6145d04-3871-a651-e7f2-397dee798ad9 Address:d6145d04-3871-a651-e7f2-397dee798ad9} {Suffrage:Voter ID:server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb Address:f65b7af9-fb9a-ed59-d7fa-56e7391d43bb} {Suffrage:Voter ID:server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d Address:d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d}]" -2024-07-05T13:29:24.790Z [INFO] server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d: entering follower state: follower="Node at d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d [Follower]" leader-address= leader-id= -2024-07-05T13:29:24.848Z [WARN] server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:24.848Z [INFO] server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb: entering candidate state: node="Node at f65b7af9-fb9a-ed59-d7fa-56e7391d43bb [Candidate]" term=3 -2024-07-05T13:29:24.848Z [INFO] server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb: election won: term=3 tally=2 -2024-07-05T13:29:24.848Z [INFO] server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb: entering leader state: leader="Node at f65b7af9-fb9a-ed59-d7fa-56e7391d43bb [Leader]" -2024-07-05T13:29:24.848Z [INFO] server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb: added peer, starting replication: peer=server-d6145d04-3871-a651-e7f2-397dee798ad9 -2024-07-05T13:29:24.848Z [INFO] server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb: added peer, starting replication: peer=server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d -2024-07-05T13:29:24.848Z [INFO] server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb: pipelining replication: peer="{Voter server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d}" -2024-07-05T13:29:24.848Z [INFO] server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb: pipelining replication: peer="{Voter server-d6145d04-3871-a651-e7f2-397dee798ad9 d6145d04-3871-a651-e7f2-397dee798ad9}" -2024-07-05T13:29:25.108Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:25.214Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:25.108574507 +0000 UTC m=+5.878485182 (1 nodes), 77.158µs from start of poll, 488.582301ms from cluster start. Timeout at 2024-07-05 13:29:25.213981769 +0000 UTC m=+5.983892444, 105.407262ms after stability -2024-07-05T13:29:25.214Z [INFO] server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb: aborting pipeline replication: peer="{Voter server-d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d d6ef9817-b430-a4cd-a885-a7b6a9ac0d2d}" -2024-07-05T13:29:25.214Z [INFO] server-f65b7af9-fb9a-ed59-d7fa-56e7391d43bb: aborting pipeline replication: peer="{Voter server-d6145d04-3871-a651-e7f2-397dee798ad9 d6145d04-3871-a651-e7f2-397dee798ad9}" +2024-07-05T13:35:39.574Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb8958c5-6fbe-d174-4c43-6ec895407c69 Address:cb8958c5-6fbe-d174-4c43-6ec895407c69} {Suffrage:Voter ID:server-72adf098-5556-20b4-b777-fda7349859b3 Address:72adf098-5556-20b4-b777-fda7349859b3} {Suffrage:Voter ID:server-b910c258-28b0-c8da-1c4b-b5a7b4c1db2c Address:b910c258-28b0-c8da-1c4b-b5a7b4c1db2c}]" +2024-07-05T13:35:39.574Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: entering follower state: follower="Node at cb8958c5-6fbe-d174-4c43-6ec895407c69 [Follower]" leader-address= leader-id= +2024-07-05T13:35:39.575Z [INFO] server-72adf098-5556-20b4-b777-fda7349859b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb8958c5-6fbe-d174-4c43-6ec895407c69 Address:cb8958c5-6fbe-d174-4c43-6ec895407c69} {Suffrage:Voter ID:server-72adf098-5556-20b4-b777-fda7349859b3 Address:72adf098-5556-20b4-b777-fda7349859b3} {Suffrage:Voter ID:server-b910c258-28b0-c8da-1c4b-b5a7b4c1db2c Address:b910c258-28b0-c8da-1c4b-b5a7b4c1db2c}]" +2024-07-05T13:35:39.575Z [INFO] server-72adf098-5556-20b4-b777-fda7349859b3: entering follower state: follower="Node at 72adf098-5556-20b4-b777-fda7349859b3 [Follower]" leader-address= leader-id= +2024-07-05T13:35:39.576Z [INFO] server-b910c258-28b0-c8da-1c4b-b5a7b4c1db2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb8958c5-6fbe-d174-4c43-6ec895407c69 Address:cb8958c5-6fbe-d174-4c43-6ec895407c69} {Suffrage:Voter ID:server-72adf098-5556-20b4-b777-fda7349859b3 Address:72adf098-5556-20b4-b777-fda7349859b3} {Suffrage:Voter ID:server-b910c258-28b0-c8da-1c4b-b5a7b4c1db2c Address:b910c258-28b0-c8da-1c4b-b5a7b4c1db2c}]" +2024-07-05T13:35:39.576Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:39.576Z [INFO] server-b910c258-28b0-c8da-1c4b-b5a7b4c1db2c: entering follower state: follower="Node at b910c258-28b0-c8da-1c4b-b5a7b4c1db2c [Follower]" leader-address= leader-id= +2024-07-05T13:35:39.645Z [WARN] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:39.645Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: entering candidate state: node="Node at cb8958c5-6fbe-d174-4c43-6ec895407c69 [Candidate]" term=2 +2024-07-05T13:35:39.646Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: election won: term=2 tally=2 +2024-07-05T13:35:39.646Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: entering leader state: leader="Node at cb8958c5-6fbe-d174-4c43-6ec895407c69 [Leader]" +2024-07-05T13:35:39.646Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: added peer, starting replication: peer=server-72adf098-5556-20b4-b777-fda7349859b3 +2024-07-05T13:35:39.646Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: added peer, starting replication: peer=server-b910c258-28b0-c8da-1c4b-b5a7b4c1db2c +2024-07-05T13:35:39.647Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: pipelining replication: peer="{Voter server-b910c258-28b0-c8da-1c4b-b5a7b4c1db2c b910c258-28b0-c8da-1c4b-b5a7b4c1db2c}" +2024-07-05T13:35:39.648Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: pipelining replication: peer="{Voter server-72adf098-5556-20b4-b777-fda7349859b3 72adf098-5556-20b4-b777-fda7349859b3}" +2024-07-05T13:35:39.753Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:39.648038991 +0000 UTC m=+5.643960443 (1 nodes), 71.013995ms from start of poll, 75.21771ms from cluster start. Timeout at 2024-07-05 13:35:39.753245611 +0000 UTC m=+5.749167062, 105.206619ms after stability +2024-07-05T13:35:39.758Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: aborting pipeline replication: peer="{Voter server-b910c258-28b0-c8da-1c4b-b5a7b4c1db2c b910c258-28b0-c8da-1c4b-b5a7b4c1db2c}" +2024-07-05T13:35:39.759Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: aborting pipeline replication: peer="{Voter server-72adf098-5556-20b4-b777-fda7349859b3 72adf098-5556-20b4-b777-fda7349859b3}" +2024-07-05T13:35:39.759Z [INFO] creating new snapshot: path=/tmp/raft1819841910/snapshots/2-6-1720186539759.tmp +2024-07-05T13:35:39.778Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: starting restore from snapshot: id=2-6-1720186539759 last-index=6 last-term=2 size-in-bytes=25 +2024-07-05T13:35:39.780Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: snapshot restore progress: id=2-6-1720186539759 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% +2024-07-05T13:35:39.780Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: restored from snapshot: id=2-6-1720186539759 last-index=6 last-term=2 size-in-bytes=25 +2024-07-05T13:35:39.780Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb8958c5-6fbe-d174-4c43-6ec895407c69 Address:cb8958c5-6fbe-d174-4c43-6ec895407c69} {Suffrage:Voter ID:server-72adf098-5556-20b4-b777-fda7349859b3 Address:72adf098-5556-20b4-b777-fda7349859b3} {Suffrage:Voter ID:server-b910c258-28b0-c8da-1c4b-b5a7b4c1db2c Address:b910c258-28b0-c8da-1c4b-b5a7b4c1db2c}]" +2024-07-05T13:35:39.781Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: entering follower state: follower="Node at cb8958c5-6fbe-d174-4c43-6ec895407c69 [Follower]" leader-address= leader-id= +2024-07-05T13:35:39.781Z [INFO] creating new snapshot: path=/tmp/raft1594588860/snapshots/2-6-1720186539781.tmp +2024-07-05T13:35:39.795Z [INFO] server-72adf098-5556-20b4-b777-fda7349859b3: starting restore from snapshot: id=2-6-1720186539781 last-index=6 last-term=2 size-in-bytes=25 +2024-07-05T13:35:39.796Z [INFO] server-72adf098-5556-20b4-b777-fda7349859b3: snapshot restore progress: id=2-6-1720186539781 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% +2024-07-05T13:35:39.796Z [INFO] server-72adf098-5556-20b4-b777-fda7349859b3: restored from snapshot: id=2-6-1720186539781 last-index=6 last-term=2 size-in-bytes=25 +2024-07-05T13:35:39.796Z [INFO] server-72adf098-5556-20b4-b777-fda7349859b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb8958c5-6fbe-d174-4c43-6ec895407c69 Address:cb8958c5-6fbe-d174-4c43-6ec895407c69} {Suffrage:Voter ID:server-72adf098-5556-20b4-b777-fda7349859b3 Address:72adf098-5556-20b4-b777-fda7349859b3} {Suffrage:Voter ID:server-b910c258-28b0-c8da-1c4b-b5a7b4c1db2c Address:b910c258-28b0-c8da-1c4b-b5a7b4c1db2c}]" +2024-07-05T13:35:39.797Z [INFO] server-72adf098-5556-20b4-b777-fda7349859b3: entering follower state: follower="Node at 72adf098-5556-20b4-b777-fda7349859b3 [Follower]" leader-address= leader-id= +2024-07-05T13:35:39.797Z [INFO] creating new snapshot: path=/tmp/raft233230667/snapshots/2-6-1720186539797.tmp +2024-07-05T13:35:39.814Z [INFO] server-b910c258-28b0-c8da-1c4b-b5a7b4c1db2c: starting restore from snapshot: id=2-6-1720186539797 last-index=6 last-term=2 size-in-bytes=25 +2024-07-05T13:35:39.816Z [INFO] server-b910c258-28b0-c8da-1c4b-b5a7b4c1db2c: snapshot restore progress: id=2-6-1720186539797 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% +2024-07-05T13:35:39.816Z [INFO] server-b910c258-28b0-c8da-1c4b-b5a7b4c1db2c: restored from snapshot: id=2-6-1720186539797 last-index=6 last-term=2 size-in-bytes=25 +2024-07-05T13:35:39.816Z [INFO] server-b910c258-28b0-c8da-1c4b-b5a7b4c1db2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb8958c5-6fbe-d174-4c43-6ec895407c69 Address:cb8958c5-6fbe-d174-4c43-6ec895407c69} {Suffrage:Voter ID:server-72adf098-5556-20b4-b777-fda7349859b3 Address:72adf098-5556-20b4-b777-fda7349859b3} {Suffrage:Voter ID:server-b910c258-28b0-c8da-1c4b-b5a7b4c1db2c Address:b910c258-28b0-c8da-1c4b-b5a7b4c1db2c}]" +2024-07-05T13:35:39.817Z [INFO] server-b910c258-28b0-c8da-1c4b-b5a7b4c1db2c: entering follower state: follower="Node at b910c258-28b0-c8da-1c4b-b5a7b4c1db2c [Follower]" leader-address= leader-id= +2024-07-05T13:35:39.862Z [WARN] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:39.862Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: entering candidate state: node="Node at cb8958c5-6fbe-d174-4c43-6ec895407c69 [Candidate]" term=3 +2024-07-05T13:35:39.863Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: election won: term=3 tally=2 +2024-07-05T13:35:39.863Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: entering leader state: leader="Node at cb8958c5-6fbe-d174-4c43-6ec895407c69 [Leader]" +2024-07-05T13:35:39.863Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: added peer, starting replication: peer=server-72adf098-5556-20b4-b777-fda7349859b3 +2024-07-05T13:35:39.864Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: added peer, starting replication: peer=server-b910c258-28b0-c8da-1c4b-b5a7b4c1db2c +2024-07-05T13:35:39.864Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: pipelining replication: peer="{Voter server-72adf098-5556-20b4-b777-fda7349859b3 72adf098-5556-20b4-b777-fda7349859b3}" +2024-07-05T13:35:39.864Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: pipelining replication: peer="{Voter server-b910c258-28b0-c8da-1c4b-b5a7b4c1db2c b910c258-28b0-c8da-1c4b-b5a7b4c1db2c}" +2024-07-05T13:35:40.132Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:40.238Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:40.132888342 +0000 UTC m=+6.128809794 (1 nodes), 233.329µs from start of poll, 560.067061ms from cluster start. Timeout at 2024-07-05 13:35:40.238409373 +0000 UTC m=+6.234330825, 105.521031ms after stability +2024-07-05T13:35:40.239Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: aborting pipeline replication: peer="{Voter server-72adf098-5556-20b4-b777-fda7349859b3 72adf098-5556-20b4-b777-fda7349859b3}" +2024-07-05T13:35:40.239Z [INFO] server-cb8958c5-6fbe-d174-4c43-6ec895407c69: aborting pipeline replication: peer="{Voter server-b910c258-28b0-c8da-1c4b-b5a7b4c1db2c b910c258-28b0-c8da-1c4b-b5a7b4c1db2c}" === RUN TestRaft_RecoverCluster/snapshot,_with_trailing_logs -2024-07-05T13:29:25.217Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84d51552-971f-e49a-f816-c90a96bd7ca6 Address:84d51552-971f-e49a-f816-c90a96bd7ca6} {Suffrage:Voter ID:server-521ea4dd-1fc9-87d8-11ef-2aea84a2e085 Address:521ea4dd-1fc9-87d8-11ef-2aea84a2e085} {Suffrage:Voter ID:server-3a2db97a-f6ad-58d5-d230-9792b48c2177 Address:3a2db97a-f6ad-58d5-d230-9792b48c2177}]" -2024-07-05T13:29:25.218Z [INFO] server-521ea4dd-1fc9-87d8-11ef-2aea84a2e085: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84d51552-971f-e49a-f816-c90a96bd7ca6 Address:84d51552-971f-e49a-f816-c90a96bd7ca6} {Suffrage:Voter ID:server-521ea4dd-1fc9-87d8-11ef-2aea84a2e085 Address:521ea4dd-1fc9-87d8-11ef-2aea84a2e085} {Suffrage:Voter ID:server-3a2db97a-f6ad-58d5-d230-9792b48c2177 Address:3a2db97a-f6ad-58d5-d230-9792b48c2177}]" -2024-07-05T13:29:25.218Z [INFO] server-3a2db97a-f6ad-58d5-d230-9792b48c2177: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84d51552-971f-e49a-f816-c90a96bd7ca6 Address:84d51552-971f-e49a-f816-c90a96bd7ca6} {Suffrage:Voter ID:server-521ea4dd-1fc9-87d8-11ef-2aea84a2e085 Address:521ea4dd-1fc9-87d8-11ef-2aea84a2e085} {Suffrage:Voter ID:server-3a2db97a-f6ad-58d5-d230-9792b48c2177 Address:3a2db97a-f6ad-58d5-d230-9792b48c2177}]" -2024-07-05T13:29:25.218Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:25.220Z [INFO] server-521ea4dd-1fc9-87d8-11ef-2aea84a2e085: entering follower state: follower="Node at 521ea4dd-1fc9-87d8-11ef-2aea84a2e085 [Follower]" leader-address= leader-id= -2024-07-05T13:29:25.220Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: entering follower state: follower="Node at 84d51552-971f-e49a-f816-c90a96bd7ca6 [Follower]" leader-address= leader-id= -2024-07-05T13:29:25.220Z [INFO] server-3a2db97a-f6ad-58d5-d230-9792b48c2177: entering follower state: follower="Node at 3a2db97a-f6ad-58d5-d230-9792b48c2177 [Follower]" leader-address= leader-id= -2024-07-05T13:29:25.304Z [WARN] server-84d51552-971f-e49a-f816-c90a96bd7ca6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:25.304Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: entering candidate state: node="Node at 84d51552-971f-e49a-f816-c90a96bd7ca6 [Candidate]" term=2 -2024-07-05T13:29:25.304Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: election won: term=2 tally=2 -2024-07-05T13:29:25.304Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: entering leader state: leader="Node at 84d51552-971f-e49a-f816-c90a96bd7ca6 [Leader]" -2024-07-05T13:29:25.304Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: added peer, starting replication: peer=server-521ea4dd-1fc9-87d8-11ef-2aea84a2e085 -2024-07-05T13:29:25.304Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: added peer, starting replication: peer=server-3a2db97a-f6ad-58d5-d230-9792b48c2177 -2024-07-05T13:29:25.304Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: pipelining replication: peer="{Voter server-3a2db97a-f6ad-58d5-d230-9792b48c2177 3a2db97a-f6ad-58d5-d230-9792b48c2177}" -2024-07-05T13:29:25.305Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: pipelining replication: peer="{Voter server-521ea4dd-1fc9-87d8-11ef-2aea84a2e085 521ea4dd-1fc9-87d8-11ef-2aea84a2e085}" -2024-07-05T13:29:25.412Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:25.305314685 +0000 UTC m=+6.075225360 (1 nodes), 86.747997ms from start of poll, 87.701563ms from cluster start. Timeout at 2024-07-05 13:29:25.412281612 +0000 UTC m=+6.182192287, 106.966927ms after stability -2024-07-05T13:29:25.419Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: aborting pipeline replication: peer="{Voter server-3a2db97a-f6ad-58d5-d230-9792b48c2177 3a2db97a-f6ad-58d5-d230-9792b48c2177}" -2024-07-05T13:29:25.419Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: aborting pipeline replication: peer="{Voter server-521ea4dd-1fc9-87d8-11ef-2aea84a2e085 521ea4dd-1fc9-87d8-11ef-2aea84a2e085}" -2024-07-05T13:29:25.420Z [INFO] creating new snapshot: path=/tmp/raft3552310448/snapshots/2-27-1720186165420.tmp -2024-07-05T13:29:25.422Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: starting restore from snapshot: id=2-27-1720186165420 last-index=27 last-term=2 size-in-bytes=168 -2024-07-05T13:29:25.422Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: snapshot restore progress: id=2-27-1720186165420 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% -2024-07-05T13:29:25.422Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: restored from snapshot: id=2-27-1720186165420 last-index=27 last-term=2 size-in-bytes=168 -2024-07-05T13:29:25.422Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84d51552-971f-e49a-f816-c90a96bd7ca6 Address:84d51552-971f-e49a-f816-c90a96bd7ca6} {Suffrage:Voter ID:server-521ea4dd-1fc9-87d8-11ef-2aea84a2e085 Address:521ea4dd-1fc9-87d8-11ef-2aea84a2e085} {Suffrage:Voter ID:server-3a2db97a-f6ad-58d5-d230-9792b48c2177 Address:3a2db97a-f6ad-58d5-d230-9792b48c2177}]" -2024-07-05T13:29:25.422Z [INFO] creating new snapshot: path=/tmp/raft144873410/snapshots/2-27-1720186165422.tmp -2024-07-05T13:29:25.423Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: entering follower state: follower="Node at 84d51552-971f-e49a-f816-c90a96bd7ca6 [Follower]" leader-address= leader-id= -2024-07-05T13:29:25.424Z [INFO] server-521ea4dd-1fc9-87d8-11ef-2aea84a2e085: starting restore from snapshot: id=2-27-1720186165422 last-index=27 last-term=2 size-in-bytes=168 -2024-07-05T13:29:25.424Z [INFO] server-521ea4dd-1fc9-87d8-11ef-2aea84a2e085: snapshot restore progress: id=2-27-1720186165422 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% -2024-07-05T13:29:25.424Z [INFO] server-521ea4dd-1fc9-87d8-11ef-2aea84a2e085: restored from snapshot: id=2-27-1720186165422 last-index=27 last-term=2 size-in-bytes=168 -2024-07-05T13:29:25.424Z [INFO] server-521ea4dd-1fc9-87d8-11ef-2aea84a2e085: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84d51552-971f-e49a-f816-c90a96bd7ca6 Address:84d51552-971f-e49a-f816-c90a96bd7ca6} {Suffrage:Voter ID:server-521ea4dd-1fc9-87d8-11ef-2aea84a2e085 Address:521ea4dd-1fc9-87d8-11ef-2aea84a2e085} {Suffrage:Voter ID:server-3a2db97a-f6ad-58d5-d230-9792b48c2177 Address:3a2db97a-f6ad-58d5-d230-9792b48c2177}]" -2024-07-05T13:29:25.424Z [INFO] creating new snapshot: path=/tmp/raft3700013015/snapshots/2-27-1720186165424.tmp -2024-07-05T13:29:25.425Z [INFO] server-521ea4dd-1fc9-87d8-11ef-2aea84a2e085: entering follower state: follower="Node at 521ea4dd-1fc9-87d8-11ef-2aea84a2e085 [Follower]" leader-address= leader-id= -2024-07-05T13:29:25.426Z [INFO] server-3a2db97a-f6ad-58d5-d230-9792b48c2177: starting restore from snapshot: id=2-27-1720186165424 last-index=27 last-term=2 size-in-bytes=168 -2024-07-05T13:29:25.426Z [INFO] server-3a2db97a-f6ad-58d5-d230-9792b48c2177: snapshot restore progress: id=2-27-1720186165424 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% -2024-07-05T13:29:25.426Z [INFO] server-3a2db97a-f6ad-58d5-d230-9792b48c2177: restored from snapshot: id=2-27-1720186165424 last-index=27 last-term=2 size-in-bytes=168 -2024-07-05T13:29:25.426Z [INFO] server-3a2db97a-f6ad-58d5-d230-9792b48c2177: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84d51552-971f-e49a-f816-c90a96bd7ca6 Address:84d51552-971f-e49a-f816-c90a96bd7ca6} {Suffrage:Voter ID:server-521ea4dd-1fc9-87d8-11ef-2aea84a2e085 Address:521ea4dd-1fc9-87d8-11ef-2aea84a2e085} {Suffrage:Voter ID:server-3a2db97a-f6ad-58d5-d230-9792b48c2177 Address:3a2db97a-f6ad-58d5-d230-9792b48c2177}]" -2024-07-05T13:29:25.426Z [INFO] server-3a2db97a-f6ad-58d5-d230-9792b48c2177: entering follower state: follower="Node at 3a2db97a-f6ad-58d5-d230-9792b48c2177 [Follower]" leader-address= leader-id= -2024-07-05T13:29:25.513Z [WARN] server-84d51552-971f-e49a-f816-c90a96bd7ca6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:25.513Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: entering candidate state: node="Node at 84d51552-971f-e49a-f816-c90a96bd7ca6 [Candidate]" term=3 -2024-07-05T13:29:25.513Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: election won: term=3 tally=2 -2024-07-05T13:29:25.513Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: entering leader state: leader="Node at 84d51552-971f-e49a-f816-c90a96bd7ca6 [Leader]" -2024-07-05T13:29:25.513Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: added peer, starting replication: peer=server-521ea4dd-1fc9-87d8-11ef-2aea84a2e085 -2024-07-05T13:29:25.513Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: added peer, starting replication: peer=server-3a2db97a-f6ad-58d5-d230-9792b48c2177 -2024-07-05T13:29:25.513Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: pipelining replication: peer="{Voter server-3a2db97a-f6ad-58d5-d230-9792b48c2177 3a2db97a-f6ad-58d5-d230-9792b48c2177}" -2024-07-05T13:29:25.514Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: pipelining replication: peer="{Voter server-521ea4dd-1fc9-87d8-11ef-2aea84a2e085 521ea4dd-1fc9-87d8-11ef-2aea84a2e085}" -2024-07-05T13:29:25.744Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:25.850Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:25.744492758 +0000 UTC m=+6.514403433 (1 nodes), 44.918µs from start of poll, 526.879636ms from cluster start. Timeout at 2024-07-05 13:29:25.850147372 +0000 UTC m=+6.620058047, 105.654614ms after stability -2024-07-05T13:29:25.850Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: aborting pipeline replication: peer="{Voter server-3a2db97a-f6ad-58d5-d230-9792b48c2177 3a2db97a-f6ad-58d5-d230-9792b48c2177}" -2024-07-05T13:29:25.850Z [INFO] server-84d51552-971f-e49a-f816-c90a96bd7ca6: aborting pipeline replication: peer="{Voter server-521ea4dd-1fc9-87d8-11ef-2aea84a2e085 521ea4dd-1fc9-87d8-11ef-2aea84a2e085}" ---- PASS: TestRaft_RecoverCluster (1.90s) - --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.67s) - --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.60s) - --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.64s) +2024-07-05T13:35:40.256Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c19ebd09-bb87-5db9-3031-fb13c308bc7a Address:c19ebd09-bb87-5db9-3031-fb13c308bc7a} {Suffrage:Voter ID:server-932605a0-9f05-7cda-967c-266a7924d81a Address:932605a0-9f05-7cda-967c-266a7924d81a} {Suffrage:Voter ID:server-c080b663-a78a-ddcc-b706-080201f5162d Address:c080b663-a78a-ddcc-b706-080201f5162d}]" +2024-07-05T13:35:40.256Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: entering follower state: follower="Node at c19ebd09-bb87-5db9-3031-fb13c308bc7a [Follower]" leader-address= leader-id= +2024-07-05T13:35:40.257Z [INFO] server-932605a0-9f05-7cda-967c-266a7924d81a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c19ebd09-bb87-5db9-3031-fb13c308bc7a Address:c19ebd09-bb87-5db9-3031-fb13c308bc7a} {Suffrage:Voter ID:server-932605a0-9f05-7cda-967c-266a7924d81a Address:932605a0-9f05-7cda-967c-266a7924d81a} {Suffrage:Voter ID:server-c080b663-a78a-ddcc-b706-080201f5162d Address:c080b663-a78a-ddcc-b706-080201f5162d}]" +2024-07-05T13:35:40.257Z [INFO] server-932605a0-9f05-7cda-967c-266a7924d81a: entering follower state: follower="Node at 932605a0-9f05-7cda-967c-266a7924d81a [Follower]" leader-address= leader-id= +2024-07-05T13:35:40.258Z [INFO] server-c080b663-a78a-ddcc-b706-080201f5162d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c19ebd09-bb87-5db9-3031-fb13c308bc7a Address:c19ebd09-bb87-5db9-3031-fb13c308bc7a} {Suffrage:Voter ID:server-932605a0-9f05-7cda-967c-266a7924d81a Address:932605a0-9f05-7cda-967c-266a7924d81a} {Suffrage:Voter ID:server-c080b663-a78a-ddcc-b706-080201f5162d Address:c080b663-a78a-ddcc-b706-080201f5162d}]" +2024-07-05T13:35:40.258Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:40.258Z [INFO] server-c080b663-a78a-ddcc-b706-080201f5162d: entering follower state: follower="Node at c080b663-a78a-ddcc-b706-080201f5162d [Follower]" leader-address= leader-id= +2024-07-05T13:35:40.342Z [WARN] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:40.342Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: entering candidate state: node="Node at c19ebd09-bb87-5db9-3031-fb13c308bc7a [Candidate]" term=2 +2024-07-05T13:35:40.343Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: election won: term=2 tally=2 +2024-07-05T13:35:40.343Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: entering leader state: leader="Node at c19ebd09-bb87-5db9-3031-fb13c308bc7a [Leader]" +2024-07-05T13:35:40.343Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: added peer, starting replication: peer=server-932605a0-9f05-7cda-967c-266a7924d81a +2024-07-05T13:35:40.343Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: added peer, starting replication: peer=server-c080b663-a78a-ddcc-b706-080201f5162d +2024-07-05T13:35:40.344Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: pipelining replication: peer="{Voter server-932605a0-9f05-7cda-967c-266a7924d81a 932605a0-9f05-7cda-967c-266a7924d81a}" +2024-07-05T13:35:40.345Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: pipelining replication: peer="{Voter server-c080b663-a78a-ddcc-b706-080201f5162d c080b663-a78a-ddcc-b706-080201f5162d}" +2024-07-05T13:35:40.449Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:40.3436927 +0000 UTC m=+6.339614152 (1 nodes), 84.98749ms from start of poll, 88.615757ms from cluster start. Timeout at 2024-07-05 13:35:40.449161815 +0000 UTC m=+6.445083267, 105.469115ms after stability +2024-07-05T13:35:40.487Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: aborting pipeline replication: peer="{Voter server-932605a0-9f05-7cda-967c-266a7924d81a 932605a0-9f05-7cda-967c-266a7924d81a}" +2024-07-05T13:35:40.487Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: aborting pipeline replication: peer="{Voter server-c080b663-a78a-ddcc-b706-080201f5162d c080b663-a78a-ddcc-b706-080201f5162d}" +2024-07-05T13:35:40.488Z [INFO] creating new snapshot: path=/tmp/raft2432813664/snapshots/2-27-1720186540488.tmp +2024-07-05T13:35:40.502Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: starting restore from snapshot: id=2-27-1720186540488 last-index=27 last-term=2 size-in-bytes=168 +2024-07-05T13:35:40.503Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: snapshot restore progress: id=2-27-1720186540488 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% +2024-07-05T13:35:40.503Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: restored from snapshot: id=2-27-1720186540488 last-index=27 last-term=2 size-in-bytes=168 +2024-07-05T13:35:40.503Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c19ebd09-bb87-5db9-3031-fb13c308bc7a Address:c19ebd09-bb87-5db9-3031-fb13c308bc7a} {Suffrage:Voter ID:server-932605a0-9f05-7cda-967c-266a7924d81a Address:932605a0-9f05-7cda-967c-266a7924d81a} {Suffrage:Voter ID:server-c080b663-a78a-ddcc-b706-080201f5162d Address:c080b663-a78a-ddcc-b706-080201f5162d}]" +2024-07-05T13:35:40.503Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: entering follower state: follower="Node at c19ebd09-bb87-5db9-3031-fb13c308bc7a [Follower]" leader-address= leader-id= +2024-07-05T13:35:40.504Z [INFO] creating new snapshot: path=/tmp/raft84816744/snapshots/2-27-1720186540504.tmp +2024-07-05T13:35:40.516Z [INFO] server-932605a0-9f05-7cda-967c-266a7924d81a: starting restore from snapshot: id=2-27-1720186540504 last-index=27 last-term=2 size-in-bytes=168 +2024-07-05T13:35:40.517Z [INFO] server-932605a0-9f05-7cda-967c-266a7924d81a: snapshot restore progress: id=2-27-1720186540504 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% +2024-07-05T13:35:40.517Z [INFO] server-932605a0-9f05-7cda-967c-266a7924d81a: restored from snapshot: id=2-27-1720186540504 last-index=27 last-term=2 size-in-bytes=168 +2024-07-05T13:35:40.517Z [INFO] server-932605a0-9f05-7cda-967c-266a7924d81a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c19ebd09-bb87-5db9-3031-fb13c308bc7a Address:c19ebd09-bb87-5db9-3031-fb13c308bc7a} {Suffrage:Voter ID:server-932605a0-9f05-7cda-967c-266a7924d81a Address:932605a0-9f05-7cda-967c-266a7924d81a} {Suffrage:Voter ID:server-c080b663-a78a-ddcc-b706-080201f5162d Address:c080b663-a78a-ddcc-b706-080201f5162d}]" +2024-07-05T13:35:40.517Z [INFO] server-932605a0-9f05-7cda-967c-266a7924d81a: entering follower state: follower="Node at 932605a0-9f05-7cda-967c-266a7924d81a [Follower]" leader-address= leader-id= +2024-07-05T13:35:40.517Z [INFO] creating new snapshot: path=/tmp/raft3535216568/snapshots/2-27-1720186540517.tmp +2024-07-05T13:35:40.531Z [INFO] server-c080b663-a78a-ddcc-b706-080201f5162d: starting restore from snapshot: id=2-27-1720186540517 last-index=27 last-term=2 size-in-bytes=168 +2024-07-05T13:35:40.531Z [INFO] server-c080b663-a78a-ddcc-b706-080201f5162d: snapshot restore progress: id=2-27-1720186540517 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% +2024-07-05T13:35:40.531Z [INFO] server-c080b663-a78a-ddcc-b706-080201f5162d: restored from snapshot: id=2-27-1720186540517 last-index=27 last-term=2 size-in-bytes=168 +2024-07-05T13:35:40.531Z [INFO] server-c080b663-a78a-ddcc-b706-080201f5162d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c19ebd09-bb87-5db9-3031-fb13c308bc7a Address:c19ebd09-bb87-5db9-3031-fb13c308bc7a} {Suffrage:Voter ID:server-932605a0-9f05-7cda-967c-266a7924d81a Address:932605a0-9f05-7cda-967c-266a7924d81a} {Suffrage:Voter ID:server-c080b663-a78a-ddcc-b706-080201f5162d Address:c080b663-a78a-ddcc-b706-080201f5162d}]" +2024-07-05T13:35:40.531Z [INFO] server-c080b663-a78a-ddcc-b706-080201f5162d: entering follower state: follower="Node at c080b663-a78a-ddcc-b706-080201f5162d [Follower]" leader-address= leader-id= +2024-07-05T13:35:40.572Z [WARN] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:40.572Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: entering candidate state: node="Node at c19ebd09-bb87-5db9-3031-fb13c308bc7a [Candidate]" term=3 +2024-07-05T13:35:40.572Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: election won: term=3 tally=2 +2024-07-05T13:35:40.572Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: entering leader state: leader="Node at c19ebd09-bb87-5db9-3031-fb13c308bc7a [Leader]" +2024-07-05T13:35:40.572Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: added peer, starting replication: peer=server-932605a0-9f05-7cda-967c-266a7924d81a +2024-07-05T13:35:40.572Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: added peer, starting replication: peer=server-c080b663-a78a-ddcc-b706-080201f5162d +2024-07-05T13:35:40.572Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: pipelining replication: peer="{Voter server-c080b663-a78a-ddcc-b706-080201f5162d c080b663-a78a-ddcc-b706-080201f5162d}" +2024-07-05T13:35:40.573Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: pipelining replication: peer="{Voter server-932605a0-9f05-7cda-967c-266a7924d81a 932605a0-9f05-7cda-967c-266a7924d81a}" +2024-07-05T13:35:40.847Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:40.953Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:40.847768659 +0000 UTC m=+6.843690110 (1 nodes), 221.662µs from start of poll, 592.691715ms from cluster start. Timeout at 2024-07-05 13:35:40.953663891 +0000 UTC m=+6.949585343, 105.895233ms after stability +2024-07-05T13:35:40.954Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: aborting pipeline replication: peer="{Voter server-932605a0-9f05-7cda-967c-266a7924d81a 932605a0-9f05-7cda-967c-266a7924d81a}" +2024-07-05T13:35:40.954Z [INFO] server-c19ebd09-bb87-5db9-3031-fb13c308bc7a: aborting pipeline replication: peer="{Voter server-c080b663-a78a-ddcc-b706-080201f5162d c080b663-a78a-ddcc-b706-080201f5162d}" +--- PASS: TestRaft_RecoverCluster (2.08s) + --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.68s) + --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.68s) + --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.71s) === RUN TestRaft_HasExistingState raft_test.go:258: Skipping unreliable test --- SKIP: TestRaft_HasExistingState (0.00s) === RUN TestRaft_SingleNode -2024-07-05T13:29:25.852Z [INFO] server-f0ee1242-a48b-d3e0-c856-55f5d433f08c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0ee1242-a48b-d3e0-c856-55f5d433f08c Address:f0ee1242-a48b-d3e0-c856-55f5d433f08c}]" -2024-07-05T13:29:25.852Z [INFO] server-f0ee1242-a48b-d3e0-c856-55f5d433f08c: entering follower state: follower="Node at f0ee1242-a48b-d3e0-c856-55f5d433f08c [Follower]" leader-address= leader-id= -2024-07-05T13:29:25.907Z [WARN] server-f0ee1242-a48b-d3e0-c856-55f5d433f08c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:25.907Z [INFO] server-f0ee1242-a48b-d3e0-c856-55f5d433f08c: entering candidate state: node="Node at f0ee1242-a48b-d3e0-c856-55f5d433f08c [Candidate]" term=2 -2024-07-05T13:29:25.907Z [INFO] server-f0ee1242-a48b-d3e0-c856-55f5d433f08c: election won: term=2 tally=1 -2024-07-05T13:29:25.907Z [INFO] server-f0ee1242-a48b-d3e0-c856-55f5d433f08c: entering leader state: leader="Node at f0ee1242-a48b-d3e0-c856-55f5d433f08c [Leader]" ---- PASS: TestRaft_SingleNode (0.06s) +2024-07-05T13:35:40.965Z [INFO] server-a7bba928-3a30-fa8d-57fd-cb2fe963ddf8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a7bba928-3a30-fa8d-57fd-cb2fe963ddf8 Address:a7bba928-3a30-fa8d-57fd-cb2fe963ddf8}]" +2024-07-05T13:35:40.965Z [INFO] server-a7bba928-3a30-fa8d-57fd-cb2fe963ddf8: entering follower state: follower="Node at a7bba928-3a30-fa8d-57fd-cb2fe963ddf8 [Follower]" leader-address= leader-id= +2024-07-05T13:35:41.041Z [WARN] server-a7bba928-3a30-fa8d-57fd-cb2fe963ddf8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:41.041Z [INFO] server-a7bba928-3a30-fa8d-57fd-cb2fe963ddf8: entering candidate state: node="Node at a7bba928-3a30-fa8d-57fd-cb2fe963ddf8 [Candidate]" term=2 +2024-07-05T13:35:41.041Z [INFO] server-a7bba928-3a30-fa8d-57fd-cb2fe963ddf8: election won: term=2 tally=1 +2024-07-05T13:35:41.041Z [INFO] server-a7bba928-3a30-fa8d-57fd-cb2fe963ddf8: entering leader state: leader="Node at a7bba928-3a30-fa8d-57fd-cb2fe963ddf8 [Leader]" +--- PASS: TestRaft_SingleNode (0.08s) === RUN TestRaft_TripleNode -2024-07-05T13:29:25.909Z [INFO] server-a99c1d9e-933f-2789-f158-9dca021d8279: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a99c1d9e-933f-2789-f158-9dca021d8279 Address:a99c1d9e-933f-2789-f158-9dca021d8279} {Suffrage:Voter ID:server-aec0192d-b44e-e274-cd83-c408dd89cdf7 Address:aec0192d-b44e-e274-cd83-c408dd89cdf7} {Suffrage:Voter ID:server-b7aa28bc-58a5-060b-1618-c830a06f95a6 Address:b7aa28bc-58a5-060b-1618-c830a06f95a6}]" -2024-07-05T13:29:25.910Z [INFO] server-aec0192d-b44e-e274-cd83-c408dd89cdf7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a99c1d9e-933f-2789-f158-9dca021d8279 Address:a99c1d9e-933f-2789-f158-9dca021d8279} {Suffrage:Voter ID:server-aec0192d-b44e-e274-cd83-c408dd89cdf7 Address:aec0192d-b44e-e274-cd83-c408dd89cdf7} {Suffrage:Voter ID:server-b7aa28bc-58a5-060b-1618-c830a06f95a6 Address:b7aa28bc-58a5-060b-1618-c830a06f95a6}]" -2024-07-05T13:29:25.910Z [INFO] server-b7aa28bc-58a5-060b-1618-c830a06f95a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a99c1d9e-933f-2789-f158-9dca021d8279 Address:a99c1d9e-933f-2789-f158-9dca021d8279} {Suffrage:Voter ID:server-aec0192d-b44e-e274-cd83-c408dd89cdf7 Address:aec0192d-b44e-e274-cd83-c408dd89cdf7} {Suffrage:Voter ID:server-b7aa28bc-58a5-060b-1618-c830a06f95a6 Address:b7aa28bc-58a5-060b-1618-c830a06f95a6}]" -2024-07-05T13:29:25.910Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:29:25.910Z [INFO] server-a99c1d9e-933f-2789-f158-9dca021d8279: entering follower state: follower="Node at a99c1d9e-933f-2789-f158-9dca021d8279 [Follower]" leader-address= leader-id= -2024-07-05T13:29:25.910Z [INFO] server-aec0192d-b44e-e274-cd83-c408dd89cdf7: entering follower state: follower="Node at aec0192d-b44e-e274-cd83-c408dd89cdf7 [Follower]" leader-address= leader-id= -2024-07-05T13:29:25.910Z [INFO] server-b7aa28bc-58a5-060b-1618-c830a06f95a6: entering follower state: follower="Node at b7aa28bc-58a5-060b-1618-c830a06f95a6 [Follower]" leader-address= leader-id= -2024-07-05T13:29:26.001Z [WARN] server-a99c1d9e-933f-2789-f158-9dca021d8279: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:26.001Z [INFO] server-a99c1d9e-933f-2789-f158-9dca021d8279: entering candidate state: node="Node at a99c1d9e-933f-2789-f158-9dca021d8279 [Candidate]" term=2 -2024-07-05T13:29:26.001Z [INFO] server-a99c1d9e-933f-2789-f158-9dca021d8279: election won: term=2 tally=2 -2024-07-05T13:29:26.001Z [INFO] server-a99c1d9e-933f-2789-f158-9dca021d8279: entering leader state: leader="Node at a99c1d9e-933f-2789-f158-9dca021d8279 [Leader]" -2024-07-05T13:29:26.001Z [INFO] server-a99c1d9e-933f-2789-f158-9dca021d8279: added peer, starting replication: peer=server-aec0192d-b44e-e274-cd83-c408dd89cdf7 -2024-07-05T13:29:26.001Z [INFO] server-a99c1d9e-933f-2789-f158-9dca021d8279: added peer, starting replication: peer=server-b7aa28bc-58a5-060b-1618-c830a06f95a6 -2024-07-05T13:29:26.002Z [INFO] server-a99c1d9e-933f-2789-f158-9dca021d8279: pipelining replication: peer="{Voter server-b7aa28bc-58a5-060b-1618-c830a06f95a6 b7aa28bc-58a5-060b-1618-c830a06f95a6}" -2024-07-05T13:29:26.002Z [INFO] server-a99c1d9e-933f-2789-f158-9dca021d8279: pipelining replication: peer="{Voter server-aec0192d-b44e-e274-cd83-c408dd89cdf7 aec0192d-b44e-e274-cd83-c408dd89cdf7}" -2024-07-05T13:29:26.108Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:29:26.002361232 +0000 UTC m=+6.772271907 (2 nodes), 92.06861ms from start of poll, 92.693748ms from cluster start. Timeout at 2024-07-05 13:29:26.108145081 +0000 UTC m=+6.878055756, 105.783849ms after stability -2024-07-05T13:29:26.108Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:26.216Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:26.108295396 +0000 UTC m=+6.878206071 (1 nodes), 1.4µs from start of poll, 198.627912ms from cluster start. Timeout at 2024-07-05 13:29:26.216429603 +0000 UTC m=+6.986340278, 108.134207ms after stability -2024-07-05T13:29:26.230Z [INFO] server-a99c1d9e-933f-2789-f158-9dca021d8279: aborting pipeline replication: peer="{Voter server-b7aa28bc-58a5-060b-1618-c830a06f95a6 b7aa28bc-58a5-060b-1618-c830a06f95a6}" -2024-07-05T13:29:26.230Z [INFO] server-a99c1d9e-933f-2789-f158-9dca021d8279: aborting pipeline replication: peer="{Voter server-aec0192d-b44e-e274-cd83-c408dd89cdf7 aec0192d-b44e-e274-cd83-c408dd89cdf7}" ---- PASS: TestRaft_TripleNode (0.32s) +2024-07-05T13:35:41.050Z [INFO] server-ae6ed429-5951-88e0-0c3b-d5f30efd5aac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ae6ed429-5951-88e0-0c3b-d5f30efd5aac Address:ae6ed429-5951-88e0-0c3b-d5f30efd5aac} {Suffrage:Voter ID:server-d2ad6f30-bc33-5f3a-dc06-3998a650d1ce Address:d2ad6f30-bc33-5f3a-dc06-3998a650d1ce} {Suffrage:Voter ID:server-784e9bff-7b4a-8d30-09fd-4396b0bcbd44 Address:784e9bff-7b4a-8d30-09fd-4396b0bcbd44}]" +2024-07-05T13:35:41.050Z [INFO] server-ae6ed429-5951-88e0-0c3b-d5f30efd5aac: entering follower state: follower="Node at ae6ed429-5951-88e0-0c3b-d5f30efd5aac [Follower]" leader-address= leader-id= +2024-07-05T13:35:41.051Z [INFO] server-d2ad6f30-bc33-5f3a-dc06-3998a650d1ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ae6ed429-5951-88e0-0c3b-d5f30efd5aac Address:ae6ed429-5951-88e0-0c3b-d5f30efd5aac} {Suffrage:Voter ID:server-d2ad6f30-bc33-5f3a-dc06-3998a650d1ce Address:d2ad6f30-bc33-5f3a-dc06-3998a650d1ce} {Suffrage:Voter ID:server-784e9bff-7b4a-8d30-09fd-4396b0bcbd44 Address:784e9bff-7b4a-8d30-09fd-4396b0bcbd44}]" +2024-07-05T13:35:41.051Z [INFO] server-d2ad6f30-bc33-5f3a-dc06-3998a650d1ce: entering follower state: follower="Node at d2ad6f30-bc33-5f3a-dc06-3998a650d1ce [Follower]" leader-address= leader-id= +2024-07-05T13:35:41.052Z [INFO] server-784e9bff-7b4a-8d30-09fd-4396b0bcbd44: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ae6ed429-5951-88e0-0c3b-d5f30efd5aac Address:ae6ed429-5951-88e0-0c3b-d5f30efd5aac} {Suffrage:Voter ID:server-d2ad6f30-bc33-5f3a-dc06-3998a650d1ce Address:d2ad6f30-bc33-5f3a-dc06-3998a650d1ce} {Suffrage:Voter ID:server-784e9bff-7b4a-8d30-09fd-4396b0bcbd44 Address:784e9bff-7b4a-8d30-09fd-4396b0bcbd44}]" +2024-07-05T13:35:41.052Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:35:41.052Z [INFO] server-784e9bff-7b4a-8d30-09fd-4396b0bcbd44: entering follower state: follower="Node at 784e9bff-7b4a-8d30-09fd-4396b0bcbd44 [Follower]" leader-address= leader-id= +2024-07-05T13:35:41.110Z [WARN] server-ae6ed429-5951-88e0-0c3b-d5f30efd5aac: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:41.110Z [INFO] server-ae6ed429-5951-88e0-0c3b-d5f30efd5aac: entering candidate state: node="Node at ae6ed429-5951-88e0-0c3b-d5f30efd5aac [Candidate]" term=2 +2024-07-05T13:35:41.110Z [INFO] server-ae6ed429-5951-88e0-0c3b-d5f30efd5aac: election won: term=2 tally=2 +2024-07-05T13:35:41.111Z [INFO] server-ae6ed429-5951-88e0-0c3b-d5f30efd5aac: entering leader state: leader="Node at ae6ed429-5951-88e0-0c3b-d5f30efd5aac [Leader]" +2024-07-05T13:35:41.111Z [INFO] server-ae6ed429-5951-88e0-0c3b-d5f30efd5aac: added peer, starting replication: peer=server-d2ad6f30-bc33-5f3a-dc06-3998a650d1ce +2024-07-05T13:35:41.111Z [INFO] server-ae6ed429-5951-88e0-0c3b-d5f30efd5aac: added peer, starting replication: peer=server-784e9bff-7b4a-8d30-09fd-4396b0bcbd44 +2024-07-05T13:35:41.112Z [INFO] server-ae6ed429-5951-88e0-0c3b-d5f30efd5aac: pipelining replication: peer="{Voter server-d2ad6f30-bc33-5f3a-dc06-3998a650d1ce d2ad6f30-bc33-5f3a-dc06-3998a650d1ce}" +2024-07-05T13:35:41.112Z [INFO] server-ae6ed429-5951-88e0-0c3b-d5f30efd5aac: pipelining replication: peer="{Voter server-784e9bff-7b4a-8d30-09fd-4396b0bcbd44 784e9bff-7b4a-8d30-09fd-4396b0bcbd44}" +2024-07-05T13:35:41.216Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:35:41.111316761 +0000 UTC m=+7.107266504 (2 nodes), 58.796593ms from start of poll, 61.599167ms from cluster start. Timeout at 2024-07-05 13:35:41.216433549 +0000 UTC m=+7.212355001, 105.088497ms after stability +2024-07-05T13:35:41.216Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:41.322Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:41.216883291 +0000 UTC m=+7.212804743 (1 nodes), 5.834µs from start of poll, 167.137406ms from cluster start. Timeout at 2024-07-05 13:35:41.322027787 +0000 UTC m=+7.317949239, 105.144496ms after stability +2024-07-05T13:35:41.334Z [INFO] server-ae6ed429-5951-88e0-0c3b-d5f30efd5aac: aborting pipeline replication: peer="{Voter server-d2ad6f30-bc33-5f3a-dc06-3998a650d1ce d2ad6f30-bc33-5f3a-dc06-3998a650d1ce}" +2024-07-05T13:35:41.334Z [INFO] server-ae6ed429-5951-88e0-0c3b-d5f30efd5aac: aborting pipeline replication: peer="{Voter server-784e9bff-7b4a-8d30-09fd-4396b0bcbd44 784e9bff-7b4a-8d30-09fd-4396b0bcbd44}" +--- PASS: TestRaft_TripleNode (0.29s) === RUN TestRaft_LeaderFail -2024-07-05T13:29:26.232Z [INFO] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 Address:ba3b19c3-f5bc-3090-c5e3-a2d59d22f634} {Suffrage:Voter ID:server-1406e6cd-858b-2f2c-48a3-bae5a0f17808 Address:1406e6cd-858b-2f2c-48a3-bae5a0f17808} {Suffrage:Voter ID:server-0a389320-05be-fd93-1314-9b7ec2ae0ead Address:0a389320-05be-fd93-1314-9b7ec2ae0ead}]" -2024-07-05T13:29:26.232Z [INFO] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 Address:ba3b19c3-f5bc-3090-c5e3-a2d59d22f634} {Suffrage:Voter ID:server-1406e6cd-858b-2f2c-48a3-bae5a0f17808 Address:1406e6cd-858b-2f2c-48a3-bae5a0f17808} {Suffrage:Voter ID:server-0a389320-05be-fd93-1314-9b7ec2ae0ead Address:0a389320-05be-fd93-1314-9b7ec2ae0ead}]" -2024-07-05T13:29:26.233Z [INFO] server-0a389320-05be-fd93-1314-9b7ec2ae0ead: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 Address:ba3b19c3-f5bc-3090-c5e3-a2d59d22f634} {Suffrage:Voter ID:server-1406e6cd-858b-2f2c-48a3-bae5a0f17808 Address:1406e6cd-858b-2f2c-48a3-bae5a0f17808} {Suffrage:Voter ID:server-0a389320-05be-fd93-1314-9b7ec2ae0ead Address:0a389320-05be-fd93-1314-9b7ec2ae0ead}]" -2024-07-05T13:29:26.233Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:29:26.233Z [INFO] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: entering follower state: follower="Node at ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 [Follower]" leader-address= leader-id= -2024-07-05T13:29:26.233Z [INFO] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: entering follower state: follower="Node at 1406e6cd-858b-2f2c-48a3-bae5a0f17808 [Follower]" leader-address= leader-id= -2024-07-05T13:29:26.233Z [INFO] server-0a389320-05be-fd93-1314-9b7ec2ae0ead: entering follower state: follower="Node at 0a389320-05be-fd93-1314-9b7ec2ae0ead [Follower]" leader-address= leader-id= -2024-07-05T13:29:26.293Z [WARN] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:26.293Z [INFO] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: entering candidate state: node="Node at 1406e6cd-858b-2f2c-48a3-bae5a0f17808 [Candidate]" term=2 -2024-07-05T13:29:26.294Z [INFO] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: election won: term=2 tally=2 -2024-07-05T13:29:26.294Z [INFO] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: entering leader state: leader="Node at 1406e6cd-858b-2f2c-48a3-bae5a0f17808 [Leader]" -2024-07-05T13:29:26.294Z [INFO] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: added peer, starting replication: peer=server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 -2024-07-05T13:29:26.294Z [INFO] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: added peer, starting replication: peer=server-0a389320-05be-fd93-1314-9b7ec2ae0ead -2024-07-05T13:29:26.294Z [INFO] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: pipelining replication: peer="{Voter server-0a389320-05be-fd93-1314-9b7ec2ae0ead 0a389320-05be-fd93-1314-9b7ec2ae0ead}" -2024-07-05T13:29:26.294Z [INFO] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: pipelining replication: peer="{Voter server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 ba3b19c3-f5bc-3090-c5e3-a2d59d22f634}" -2024-07-05T13:29:26.400Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:29:26.294711736 +0000 UTC m=+7.064622411 (2 nodes), 61.55652ms from start of poll, 62.275575ms from cluster start. Timeout at 2024-07-05 13:29:26.400221715 +0000 UTC m=+7.170132390, 105.509979ms after stability -2024-07-05T13:29:26.400Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:26.506Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:26.400444427 +0000 UTC m=+7.170355102 (1 nodes), 2µs from start of poll, 168.008266ms from cluster start. Timeout at 2024-07-05 13:29:26.506143279 +0000 UTC m=+7.276053954, 105.698852ms after stability - raft_test.go:378: [INFO] Disconnecting Node at 1406e6cd-858b-2f2c-48a3-bae5a0f17808 [Leader] -2024-07-05T13:29:26.519Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to pipeline appendEntries: peer="{Voter server-0a389320-05be-fd93-1314-9b7ec2ae0ead 0a389320-05be-fd93-1314-9b7ec2ae0ead}" error="append pipeline closed" -2024-07-05T13:29:26.519Z [INFO] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: aborting pipeline replication: peer="{Voter server-0a389320-05be-fd93-1314-9b7ec2ae0ead 0a389320-05be-fd93-1314-9b7ec2ae0ead}" -2024-07-05T13:29:26.521Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to heartbeat to: peer=ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 backoff time=10ms error="failed to connect to peer: ba3b19c3-f5bc-3090-c5e3-a2d59d22f634" -2024-07-05T13:29:26.521Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:26.525Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to pipeline appendEntries: peer="{Voter server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 ba3b19c3-f5bc-3090-c5e3-a2d59d22f634}" error="append pipeline closed" -2024-07-05T13:29:26.525Z [INFO] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: aborting pipeline replication: peer="{Voter server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 ba3b19c3-f5bc-3090-c5e3-a2d59d22f634}" -2024-07-05T13:29:26.526Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to appendEntries to: peer="{Voter server-0a389320-05be-fd93-1314-9b7ec2ae0ead 0a389320-05be-fd93-1314-9b7ec2ae0ead}" error="failed to connect to peer: 0a389320-05be-fd93-1314-9b7ec2ae0ead" -2024-07-05T13:29:26.529Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to heartbeat to: peer=0a389320-05be-fd93-1314-9b7ec2ae0ead backoff time=10ms error="failed to connect to peer: 0a389320-05be-fd93-1314-9b7ec2ae0ead" -2024-07-05T13:29:26.533Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to appendEntries to: peer="{Voter server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 ba3b19c3-f5bc-3090-c5e3-a2d59d22f634}" error="failed to connect to peer: ba3b19c3-f5bc-3090-c5e3-a2d59d22f634" -2024-07-05T13:29:26.538Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to heartbeat to: peer=ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 backoff time=10ms error="failed to connect to peer: ba3b19c3-f5bc-3090-c5e3-a2d59d22f634" -2024-07-05T13:29:26.547Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to appendEntries to: peer="{Voter server-0a389320-05be-fd93-1314-9b7ec2ae0ead 0a389320-05be-fd93-1314-9b7ec2ae0ead}" error="failed to connect to peer: 0a389320-05be-fd93-1314-9b7ec2ae0ead" -2024-07-05T13:29:26.548Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to heartbeat to: peer=0a389320-05be-fd93-1314-9b7ec2ae0ead backoff time=10ms error="failed to connect to peer: 0a389320-05be-fd93-1314-9b7ec2ae0ead" -2024-07-05T13:29:26.554Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to appendEntries to: peer="{Voter server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 ba3b19c3-f5bc-3090-c5e3-a2d59d22f634}" error="failed to connect to peer: ba3b19c3-f5bc-3090-c5e3-a2d59d22f634" -2024-07-05T13:29:26.557Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to heartbeat to: peer=ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 backoff time=10ms error="failed to connect to peer: ba3b19c3-f5bc-3090-c5e3-a2d59d22f634" -2024-07-05T13:29:26.566Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to appendEntries to: peer="{Voter server-0a389320-05be-fd93-1314-9b7ec2ae0ead 0a389320-05be-fd93-1314-9b7ec2ae0ead}" error="failed to connect to peer: 0a389320-05be-fd93-1314-9b7ec2ae0ead" -2024-07-05T13:29:26.566Z [WARN] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to contact: server-id=server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 time=50.471029ms -2024-07-05T13:29:26.568Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to heartbeat to: peer=0a389320-05be-fd93-1314-9b7ec2ae0ead backoff time=10ms error="failed to connect to peer: 0a389320-05be-fd93-1314-9b7ec2ae0ead" -2024-07-05T13:29:26.573Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to appendEntries to: peer="{Voter server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 ba3b19c3-f5bc-3090-c5e3-a2d59d22f634}" error="failed to connect to peer: ba3b19c3-f5bc-3090-c5e3-a2d59d22f634" -2024-07-05T13:29:26.576Z [WARN] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to contact: server-id=server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 time=60.665112ms -2024-07-05T13:29:26.576Z [WARN] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to contact: server-id=server-0a389320-05be-fd93-1314-9b7ec2ae0ead time=58.322274ms -2024-07-05T13:29:26.577Z [WARN] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to contact quorum of nodes, stepping down -2024-07-05T13:29:26.577Z [INFO] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: entering follower state: follower="Node at 1406e6cd-858b-2f2c-48a3-bae5a0f17808 [Follower]" leader-address= leader-id= -2024-07-05T13:29:26.587Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to heartbeat to: peer=0a389320-05be-fd93-1314-9b7ec2ae0ead backoff time=20ms error="failed to connect to peer: 0a389320-05be-fd93-1314-9b7ec2ae0ead" -2024-07-05T13:29:26.587Z [WARN] server-0a389320-05be-fd93-1314-9b7ec2ae0ead: heartbeat timeout reached, starting election: last-leader-addr=1406e6cd-858b-2f2c-48a3-bae5a0f17808 last-leader-id=server-1406e6cd-858b-2f2c-48a3-bae5a0f17808 -2024-07-05T13:29:26.587Z [INFO] server-0a389320-05be-fd93-1314-9b7ec2ae0ead: entering candidate state: node="Node at 0a389320-05be-fd93-1314-9b7ec2ae0ead [Candidate]" term=3 -2024-07-05T13:29:26.587Z [ERROR] server-0a389320-05be-fd93-1314-9b7ec2ae0ead: failed to make requestVote RPC: target="{Voter server-1406e6cd-858b-2f2c-48a3-bae5a0f17808 1406e6cd-858b-2f2c-48a3-bae5a0f17808}" error="failed to connect to peer: 1406e6cd-858b-2f2c-48a3-bae5a0f17808" term=3 -2024-07-05T13:29:26.587Z [WARN] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: rejecting vote request since we have a leader: from=0a389320-05be-fd93-1314-9b7ec2ae0ead leader=1406e6cd-858b-2f2c-48a3-bae5a0f17808 leader-id=server-1406e6cd-858b-2f2c-48a3-bae5a0f17808 -2024-07-05T13:29:26.596Z [WARN] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: heartbeat timeout reached, starting election: last-leader-addr=1406e6cd-858b-2f2c-48a3-bae5a0f17808 last-leader-id=server-1406e6cd-858b-2f2c-48a3-bae5a0f17808 -2024-07-05T13:29:26.596Z [INFO] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: entering candidate state: node="Node at ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 [Candidate]" term=3 -2024-07-05T13:29:26.596Z [INFO] server-0a389320-05be-fd93-1314-9b7ec2ae0ead: duplicate requestVote for same term: term=3 -2024-07-05T13:29:26.596Z [ERROR] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: failed to make requestVote RPC: target="{Voter server-1406e6cd-858b-2f2c-48a3-bae5a0f17808 1406e6cd-858b-2f2c-48a3-bae5a0f17808}" error="failed to connect to peer: 1406e6cd-858b-2f2c-48a3-bae5a0f17808" term=3 -2024-07-05T13:29:26.596Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to appendEntries to: peer="{Voter server-0a389320-05be-fd93-1314-9b7ec2ae0ead 0a389320-05be-fd93-1314-9b7ec2ae0ead}" error="failed to connect to peer: 0a389320-05be-fd93-1314-9b7ec2ae0ead" -2024-07-05T13:29:26.636Z [WARN] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:26.636Z [INFO] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: entering candidate state: node="Node at 1406e6cd-858b-2f2c-48a3-bae5a0f17808 [Candidate]" term=3 -2024-07-05T13:29:26.636Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to make requestVote RPC: target="{Voter server-0a389320-05be-fd93-1314-9b7ec2ae0ead 0a389320-05be-fd93-1314-9b7ec2ae0ead}" error="failed to connect to peer: 0a389320-05be-fd93-1314-9b7ec2ae0ead" term=3 -2024-07-05T13:29:26.636Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to make requestVote RPC: target="{Voter server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 ba3b19c3-f5bc-3090-c5e3-a2d59d22f634}" error="failed to connect to peer: ba3b19c3-f5bc-3090-c5e3-a2d59d22f634" term=3 -2024-07-05T13:29:26.656Z [WARN] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: Election timeout reached, restarting election -2024-07-05T13:29:26.656Z [INFO] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: entering candidate state: node="Node at ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 [Candidate]" term=4 -2024-07-05T13:29:26.656Z [INFO] server-0a389320-05be-fd93-1314-9b7ec2ae0ead: entering follower state: follower="Node at 0a389320-05be-fd93-1314-9b7ec2ae0ead [Follower]" leader-address= leader-id= -2024-07-05T13:29:26.656Z [INFO] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: election won: term=4 tally=2 -2024-07-05T13:29:26.656Z [INFO] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: entering leader state: leader="Node at ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 [Leader]" -2024-07-05T13:29:26.656Z [INFO] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: added peer, starting replication: peer=server-1406e6cd-858b-2f2c-48a3-bae5a0f17808 -2024-07-05T13:29:26.656Z [INFO] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: added peer, starting replication: peer=server-0a389320-05be-fd93-1314-9b7ec2ae0ead -2024-07-05T13:29:26.657Z [INFO] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: pipelining replication: peer="{Voter server-0a389320-05be-fd93-1314-9b7ec2ae0ead 0a389320-05be-fd93-1314-9b7ec2ae0ead}" -2024-07-05T13:29:26.657Z [ERROR] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: failed to make requestVote RPC: target="{Voter server-1406e6cd-858b-2f2c-48a3-bae5a0f17808 1406e6cd-858b-2f2c-48a3-bae5a0f17808}" error="failed to connect to peer: 1406e6cd-858b-2f2c-48a3-bae5a0f17808" term=4 -2024-07-05T13:29:26.657Z [ERROR] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: failed to appendEntries to: peer="{Voter server-1406e6cd-858b-2f2c-48a3-bae5a0f17808 1406e6cd-858b-2f2c-48a3-bae5a0f17808}" error="failed to connect to peer: 1406e6cd-858b-2f2c-48a3-bae5a0f17808" -2024-07-05T13:29:26.664Z [ERROR] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: failed to heartbeat to: peer=1406e6cd-858b-2f2c-48a3-bae5a0f17808 backoff time=10ms error="failed to connect to peer: 1406e6cd-858b-2f2c-48a3-bae5a0f17808" -2024-07-05T13:29:26.678Z [ERROR] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: failed to appendEntries to: peer="{Voter server-1406e6cd-858b-2f2c-48a3-bae5a0f17808 1406e6cd-858b-2f2c-48a3-bae5a0f17808}" error="failed to connect to peer: 1406e6cd-858b-2f2c-48a3-bae5a0f17808" -2024-07-05T13:29:26.682Z [ERROR] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: failed to heartbeat to: peer=1406e6cd-858b-2f2c-48a3-bae5a0f17808 backoff time=10ms error="failed to connect to peer: 1406e6cd-858b-2f2c-48a3-bae5a0f17808" -2024-07-05T13:29:26.695Z [ERROR] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: failed to appendEntries to: peer="{Voter server-1406e6cd-858b-2f2c-48a3-bae5a0f17808 1406e6cd-858b-2f2c-48a3-bae5a0f17808}" error="failed to connect to peer: 1406e6cd-858b-2f2c-48a3-bae5a0f17808" -2024-07-05T13:29:26.699Z [ERROR] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: failed to heartbeat to: peer=1406e6cd-858b-2f2c-48a3-bae5a0f17808 backoff time=10ms error="failed to connect to peer: 1406e6cd-858b-2f2c-48a3-bae5a0f17808" -2024-07-05T13:29:26.707Z [WARN] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: failed to contact: server-id=server-1406e6cd-858b-2f2c-48a3-bae5a0f17808 time=50.16328ms -2024-07-05T13:29:26.716Z [WARN] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: Election timeout reached, restarting election -2024-07-05T13:29:26.716Z [INFO] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: entering candidate state: node="Node at 1406e6cd-858b-2f2c-48a3-bae5a0f17808 [Candidate]" term=4 -2024-07-05T13:29:26.716Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to make requestVote RPC: target="{Voter server-0a389320-05be-fd93-1314-9b7ec2ae0ead 0a389320-05be-fd93-1314-9b7ec2ae0ead}" error="failed to connect to peer: 0a389320-05be-fd93-1314-9b7ec2ae0ead" term=4 -2024-07-05T13:29:26.716Z [ERROR] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: failed to make requestVote RPC: target="{Voter server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 ba3b19c3-f5bc-3090-c5e3-a2d59d22f634}" error="failed to connect to peer: ba3b19c3-f5bc-3090-c5e3-a2d59d22f634" term=4 -2024-07-05T13:29:26.718Z [ERROR] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: failed to heartbeat to: peer=1406e6cd-858b-2f2c-48a3-bae5a0f17808 backoff time=20ms error="failed to connect to peer: 1406e6cd-858b-2f2c-48a3-bae5a0f17808" -2024-07-05T13:29:26.725Z [ERROR] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: failed to appendEntries to: peer="{Voter server-1406e6cd-858b-2f2c-48a3-bae5a0f17808 1406e6cd-858b-2f2c-48a3-bae5a0f17808}" error="failed to connect to peer: 1406e6cd-858b-2f2c-48a3-bae5a0f17808" -2024-07-05T13:29:26.746Z [ERROR] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: failed to heartbeat to: peer=1406e6cd-858b-2f2c-48a3-bae5a0f17808 backoff time=25ms error="failed to connect to peer: 1406e6cd-858b-2f2c-48a3-bae5a0f17808" -2024-07-05T13:29:26.752Z [WARN] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: failed to contact: server-id=server-1406e6cd-858b-2f2c-48a3-bae5a0f17808 time=96.103669ms -2024-07-05T13:29:26.763Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:26.657235139 +0000 UTC m=+7.427145814 (1 nodes), 136.008429ms from start of poll, 424.798978ms from cluster start. Timeout at 2024-07-05 13:29:26.762985469 +0000 UTC m=+7.532896184, 105.75037ms after stability - raft_test.go:413: [INFO] Reconnecting Node at 1406e6cd-858b-2f2c-48a3-bae5a0f17808 [Candidate] -2024-07-05T13:29:26.772Z [INFO] server-1406e6cd-858b-2f2c-48a3-bae5a0f17808: entering follower state: follower="Node at 1406e6cd-858b-2f2c-48a3-bae5a0f17808 [Follower]" leader-address=ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 leader-id=server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634 -2024-07-05T13:29:26.772Z [INFO] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: pipelining replication: peer="{Voter server-1406e6cd-858b-2f2c-48a3-bae5a0f17808 1406e6cd-858b-2f2c-48a3-bae5a0f17808}" -2024-07-05T13:29:26.778Z [INFO] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: aborting pipeline replication: peer="{Voter server-0a389320-05be-fd93-1314-9b7ec2ae0ead 0a389320-05be-fd93-1314-9b7ec2ae0ead}" -2024-07-05T13:29:26.778Z [INFO] server-ba3b19c3-f5bc-3090-c5e3-a2d59d22f634: aborting pipeline replication: peer="{Voter server-1406e6cd-858b-2f2c-48a3-bae5a0f17808 1406e6cd-858b-2f2c-48a3-bae5a0f17808}" ---- PASS: TestRaft_LeaderFail (0.55s) +2024-07-05T13:35:41.345Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cd44dc2-28eb-71b8-06e8-1728a102716c Address:0cd44dc2-28eb-71b8-06e8-1728a102716c} {Suffrage:Voter ID:server-3c146ef7-4a33-8738-718b-24adb7fd9c7e Address:3c146ef7-4a33-8738-718b-24adb7fd9c7e} {Suffrage:Voter ID:server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 Address:d6fd6f1b-0f43-ced8-1bb3-379a603e4079}]" +2024-07-05T13:35:41.345Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: entering follower state: follower="Node at 0cd44dc2-28eb-71b8-06e8-1728a102716c [Follower]" leader-address= leader-id= +2024-07-05T13:35:41.346Z [INFO] server-3c146ef7-4a33-8738-718b-24adb7fd9c7e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cd44dc2-28eb-71b8-06e8-1728a102716c Address:0cd44dc2-28eb-71b8-06e8-1728a102716c} {Suffrage:Voter ID:server-3c146ef7-4a33-8738-718b-24adb7fd9c7e Address:3c146ef7-4a33-8738-718b-24adb7fd9c7e} {Suffrage:Voter ID:server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 Address:d6fd6f1b-0f43-ced8-1bb3-379a603e4079}]" +2024-07-05T13:35:41.346Z [INFO] server-3c146ef7-4a33-8738-718b-24adb7fd9c7e: entering follower state: follower="Node at 3c146ef7-4a33-8738-718b-24adb7fd9c7e [Follower]" leader-address= leader-id= +2024-07-05T13:35:41.347Z [INFO] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cd44dc2-28eb-71b8-06e8-1728a102716c Address:0cd44dc2-28eb-71b8-06e8-1728a102716c} {Suffrage:Voter ID:server-3c146ef7-4a33-8738-718b-24adb7fd9c7e Address:3c146ef7-4a33-8738-718b-24adb7fd9c7e} {Suffrage:Voter ID:server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 Address:d6fd6f1b-0f43-ced8-1bb3-379a603e4079}]" +2024-07-05T13:35:41.347Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:35:41.347Z [INFO] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: entering follower state: follower="Node at d6fd6f1b-0f43-ced8-1bb3-379a603e4079 [Follower]" leader-address= leader-id= +2024-07-05T13:35:41.407Z [WARN] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:41.407Z [INFO] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: entering candidate state: node="Node at d6fd6f1b-0f43-ced8-1bb3-379a603e4079 [Candidate]" term=2 +2024-07-05T13:35:41.407Z [INFO] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: election won: term=2 tally=2 +2024-07-05T13:35:41.407Z [INFO] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: entering leader state: leader="Node at d6fd6f1b-0f43-ced8-1bb3-379a603e4079 [Leader]" +2024-07-05T13:35:41.408Z [INFO] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: added peer, starting replication: peer=server-0cd44dc2-28eb-71b8-06e8-1728a102716c +2024-07-05T13:35:41.408Z [INFO] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: added peer, starting replication: peer=server-3c146ef7-4a33-8738-718b-24adb7fd9c7e +2024-07-05T13:35:41.408Z [INFO] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: pipelining replication: peer="{Voter server-0cd44dc2-28eb-71b8-06e8-1728a102716c 0cd44dc2-28eb-71b8-06e8-1728a102716c}" +2024-07-05T13:35:41.409Z [INFO] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: pipelining replication: peer="{Voter server-3c146ef7-4a33-8738-718b-24adb7fd9c7e 3c146ef7-4a33-8738-718b-24adb7fd9c7e}" +2024-07-05T13:35:41.513Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:35:41.408123589 +0000 UTC m=+7.404045333 (2 nodes), 60.177318ms from start of poll, 63.737628ms from cluster start. Timeout at 2024-07-05 13:35:41.513432291 +0000 UTC m=+7.509353742, 105.308409ms after stability +2024-07-05T13:35:41.513Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:41.619Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:41.513838283 +0000 UTC m=+7.509759735 (1 nodes), 6.708µs from start of poll, 169.45203ms from cluster start. Timeout at 2024-07-05 13:35:41.618965571 +0000 UTC m=+7.614887023, 105.127288ms after stability + raft_test.go:378: [INFO] Disconnecting Node at d6fd6f1b-0f43-ced8-1bb3-379a603e4079 [Leader] +2024-07-05T13:35:41.634Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to heartbeat to: peer=0cd44dc2-28eb-71b8-06e8-1728a102716c backoff time=10ms error="failed to connect to peer: 0cd44dc2-28eb-71b8-06e8-1728a102716c" +2024-07-05T13:35:41.634Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:41.636Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to pipeline appendEntries: peer="{Voter server-3c146ef7-4a33-8738-718b-24adb7fd9c7e 3c146ef7-4a33-8738-718b-24adb7fd9c7e}" error="append pipeline closed" +2024-07-05T13:35:41.636Z [INFO] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: aborting pipeline replication: peer="{Voter server-3c146ef7-4a33-8738-718b-24adb7fd9c7e 3c146ef7-4a33-8738-718b-24adb7fd9c7e}" +2024-07-05T13:35:41.637Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to pipeline appendEntries: peer="{Voter server-0cd44dc2-28eb-71b8-06e8-1728a102716c 0cd44dc2-28eb-71b8-06e8-1728a102716c}" error="append pipeline closed" +2024-07-05T13:35:41.637Z [INFO] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: aborting pipeline replication: peer="{Voter server-0cd44dc2-28eb-71b8-06e8-1728a102716c 0cd44dc2-28eb-71b8-06e8-1728a102716c}" +2024-07-05T13:35:41.639Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to heartbeat to: peer=3c146ef7-4a33-8738-718b-24adb7fd9c7e backoff time=10ms error="failed to connect to peer: 3c146ef7-4a33-8738-718b-24adb7fd9c7e" +2024-07-05T13:35:41.646Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to appendEntries to: peer="{Voter server-3c146ef7-4a33-8738-718b-24adb7fd9c7e 3c146ef7-4a33-8738-718b-24adb7fd9c7e}" error="failed to connect to peer: 3c146ef7-4a33-8738-718b-24adb7fd9c7e" +2024-07-05T13:35:41.646Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to appendEntries to: peer="{Voter server-0cd44dc2-28eb-71b8-06e8-1728a102716c 0cd44dc2-28eb-71b8-06e8-1728a102716c}" error="failed to connect to peer: 0cd44dc2-28eb-71b8-06e8-1728a102716c" +2024-07-05T13:35:41.654Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to heartbeat to: peer=0cd44dc2-28eb-71b8-06e8-1728a102716c backoff time=10ms error="failed to connect to peer: 0cd44dc2-28eb-71b8-06e8-1728a102716c" +2024-07-05T13:35:41.655Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to heartbeat to: peer=3c146ef7-4a33-8738-718b-24adb7fd9c7e backoff time=10ms error="failed to connect to peer: 3c146ef7-4a33-8738-718b-24adb7fd9c7e" +2024-07-05T13:35:41.664Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to appendEntries to: peer="{Voter server-3c146ef7-4a33-8738-718b-24adb7fd9c7e 3c146ef7-4a33-8738-718b-24adb7fd9c7e}" error="failed to connect to peer: 3c146ef7-4a33-8738-718b-24adb7fd9c7e" +2024-07-05T13:35:41.666Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to appendEntries to: peer="{Voter server-0cd44dc2-28eb-71b8-06e8-1728a102716c 0cd44dc2-28eb-71b8-06e8-1728a102716c}" error="failed to connect to peer: 0cd44dc2-28eb-71b8-06e8-1728a102716c" +2024-07-05T13:35:41.670Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to heartbeat to: peer=0cd44dc2-28eb-71b8-06e8-1728a102716c backoff time=10ms error="failed to connect to peer: 0cd44dc2-28eb-71b8-06e8-1728a102716c" +2024-07-05T13:35:41.676Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to heartbeat to: peer=3c146ef7-4a33-8738-718b-24adb7fd9c7e backoff time=10ms error="failed to connect to peer: 3c146ef7-4a33-8738-718b-24adb7fd9c7e" +2024-07-05T13:35:41.680Z [WARN] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to contact: server-id=server-0cd44dc2-28eb-71b8-06e8-1728a102716c time=50.230208ms +2024-07-05T13:35:41.683Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to appendEntries to: peer="{Voter server-3c146ef7-4a33-8738-718b-24adb7fd9c7e 3c146ef7-4a33-8738-718b-24adb7fd9c7e}" error="failed to connect to peer: 3c146ef7-4a33-8738-718b-24adb7fd9c7e" +2024-07-05T13:35:41.686Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to appendEntries to: peer="{Voter server-0cd44dc2-28eb-71b8-06e8-1728a102716c 0cd44dc2-28eb-71b8-06e8-1728a102716c}" error="failed to connect to peer: 0cd44dc2-28eb-71b8-06e8-1728a102716c" +2024-07-05T13:35:41.689Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to heartbeat to: peer=0cd44dc2-28eb-71b8-06e8-1728a102716c backoff time=20ms error="failed to connect to peer: 0cd44dc2-28eb-71b8-06e8-1728a102716c" +2024-07-05T13:35:41.691Z [WARN] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to contact: server-id=server-0cd44dc2-28eb-71b8-06e8-1728a102716c time=60.993094ms +2024-07-05T13:35:41.691Z [WARN] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to contact: server-id=server-3c146ef7-4a33-8738-718b-24adb7fd9c7e time=60.569018ms +2024-07-05T13:35:41.691Z [WARN] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to contact quorum of nodes, stepping down +2024-07-05T13:35:41.691Z [INFO] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: entering follower state: follower="Node at d6fd6f1b-0f43-ced8-1bb3-379a603e4079 [Follower]" leader-address= leader-id= +2024-07-05T13:35:41.695Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to heartbeat to: peer=3c146ef7-4a33-8738-718b-24adb7fd9c7e backoff time=20ms error="failed to connect to peer: 3c146ef7-4a33-8738-718b-24adb7fd9c7e" +2024-07-05T13:35:41.704Z [WARN] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: heartbeat timeout reached, starting election: last-leader-addr=d6fd6f1b-0f43-ced8-1bb3-379a603e4079 last-leader-id=server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 +2024-07-05T13:35:41.704Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: entering candidate state: node="Node at 0cd44dc2-28eb-71b8-06e8-1728a102716c [Candidate]" term=3 +2024-07-05T13:35:41.705Z [WARN] server-3c146ef7-4a33-8738-718b-24adb7fd9c7e: rejecting vote request since we have a leader: from=0cd44dc2-28eb-71b8-06e8-1728a102716c leader=d6fd6f1b-0f43-ced8-1bb3-379a603e4079 leader-id=server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 +2024-07-05T13:35:41.705Z [ERROR] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: failed to make requestVote RPC: target="{Voter server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 d6fd6f1b-0f43-ced8-1bb3-379a603e4079}" error="failed to connect to peer: d6fd6f1b-0f43-ced8-1bb3-379a603e4079" term=3 +2024-07-05T13:35:41.710Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to appendEntries to: peer="{Voter server-3c146ef7-4a33-8738-718b-24adb7fd9c7e 3c146ef7-4a33-8738-718b-24adb7fd9c7e}" error="failed to connect to peer: 3c146ef7-4a33-8738-718b-24adb7fd9c7e" +2024-07-05T13:35:41.743Z [WARN] server-3c146ef7-4a33-8738-718b-24adb7fd9c7e: heartbeat timeout reached, starting election: last-leader-addr=d6fd6f1b-0f43-ced8-1bb3-379a603e4079 last-leader-id=server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 +2024-07-05T13:35:41.743Z [INFO] server-3c146ef7-4a33-8738-718b-24adb7fd9c7e: entering candidate state: node="Node at 3c146ef7-4a33-8738-718b-24adb7fd9c7e [Candidate]" term=3 +2024-07-05T13:35:41.744Z [ERROR] server-3c146ef7-4a33-8738-718b-24adb7fd9c7e: failed to make requestVote RPC: target="{Voter server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 d6fd6f1b-0f43-ced8-1bb3-379a603e4079}" error="failed to connect to peer: d6fd6f1b-0f43-ced8-1bb3-379a603e4079" term=3 +2024-07-05T13:35:41.744Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: duplicate requestVote for same term: term=3 +2024-07-05T13:35:41.751Z [WARN] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:41.751Z [INFO] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: entering candidate state: node="Node at d6fd6f1b-0f43-ced8-1bb3-379a603e4079 [Candidate]" term=3 +2024-07-05T13:35:41.752Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to make requestVote RPC: target="{Voter server-3c146ef7-4a33-8738-718b-24adb7fd9c7e 3c146ef7-4a33-8738-718b-24adb7fd9c7e}" error="failed to connect to peer: 3c146ef7-4a33-8738-718b-24adb7fd9c7e" term=3 +2024-07-05T13:35:41.752Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to make requestVote RPC: target="{Voter server-0cd44dc2-28eb-71b8-06e8-1728a102716c 0cd44dc2-28eb-71b8-06e8-1728a102716c}" error="failed to connect to peer: 0cd44dc2-28eb-71b8-06e8-1728a102716c" term=3 +2024-07-05T13:35:41.777Z [WARN] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: Election timeout reached, restarting election +2024-07-05T13:35:41.777Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: entering candidate state: node="Node at 0cd44dc2-28eb-71b8-06e8-1728a102716c [Candidate]" term=4 +2024-07-05T13:35:41.778Z [ERROR] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: failed to make requestVote RPC: target="{Voter server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 d6fd6f1b-0f43-ced8-1bb3-379a603e4079}" error="failed to connect to peer: d6fd6f1b-0f43-ced8-1bb3-379a603e4079" term=4 +2024-07-05T13:35:41.778Z [INFO] server-3c146ef7-4a33-8738-718b-24adb7fd9c7e: entering follower state: follower="Node at 3c146ef7-4a33-8738-718b-24adb7fd9c7e [Follower]" leader-address= leader-id= +2024-07-05T13:35:41.778Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: election won: term=4 tally=2 +2024-07-05T13:35:41.778Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: entering leader state: leader="Node at 0cd44dc2-28eb-71b8-06e8-1728a102716c [Leader]" +2024-07-05T13:35:41.779Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: added peer, starting replication: peer=server-3c146ef7-4a33-8738-718b-24adb7fd9c7e +2024-07-05T13:35:41.779Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: added peer, starting replication: peer=server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 +2024-07-05T13:35:41.779Z [ERROR] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: failed to appendEntries to: peer="{Voter server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 d6fd6f1b-0f43-ced8-1bb3-379a603e4079}" error="failed to connect to peer: d6fd6f1b-0f43-ced8-1bb3-379a603e4079" +2024-07-05T13:35:41.779Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: pipelining replication: peer="{Voter server-3c146ef7-4a33-8738-718b-24adb7fd9c7e 3c146ef7-4a33-8738-718b-24adb7fd9c7e}" +2024-07-05T13:35:41.785Z [ERROR] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: failed to heartbeat to: peer=d6fd6f1b-0f43-ced8-1bb3-379a603e4079 backoff time=10ms error="failed to connect to peer: d6fd6f1b-0f43-ced8-1bb3-379a603e4079" +2024-07-05T13:35:41.800Z [ERROR] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: failed to appendEntries to: peer="{Voter server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 d6fd6f1b-0f43-ced8-1bb3-379a603e4079}" error="failed to connect to peer: d6fd6f1b-0f43-ced8-1bb3-379a603e4079" +2024-07-05T13:35:41.804Z [ERROR] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: failed to heartbeat to: peer=d6fd6f1b-0f43-ced8-1bb3-379a603e4079 backoff time=10ms error="failed to connect to peer: d6fd6f1b-0f43-ced8-1bb3-379a603e4079" +2024-07-05T13:35:41.818Z [ERROR] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: failed to appendEntries to: peer="{Voter server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 d6fd6f1b-0f43-ced8-1bb3-379a603e4079}" error="failed to connect to peer: d6fd6f1b-0f43-ced8-1bb3-379a603e4079" +2024-07-05T13:35:41.820Z [WARN] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: Election timeout reached, restarting election +2024-07-05T13:35:41.820Z [INFO] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: entering candidate state: node="Node at d6fd6f1b-0f43-ced8-1bb3-379a603e4079 [Candidate]" term=4 +2024-07-05T13:35:41.820Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to make requestVote RPC: target="{Voter server-0cd44dc2-28eb-71b8-06e8-1728a102716c 0cd44dc2-28eb-71b8-06e8-1728a102716c}" error="failed to connect to peer: 0cd44dc2-28eb-71b8-06e8-1728a102716c" term=4 +2024-07-05T13:35:41.820Z [ERROR] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to make requestVote RPC: target="{Voter server-3c146ef7-4a33-8738-718b-24adb7fd9c7e 3c146ef7-4a33-8738-718b-24adb7fd9c7e}" error="failed to connect to peer: 3c146ef7-4a33-8738-718b-24adb7fd9c7e" term=4 +2024-07-05T13:35:41.824Z [ERROR] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: failed to heartbeat to: peer=d6fd6f1b-0f43-ced8-1bb3-379a603e4079 backoff time=10ms error="failed to connect to peer: d6fd6f1b-0f43-ced8-1bb3-379a603e4079" +2024-07-05T13:35:41.829Z [WARN] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: failed to contact: server-id=server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 time=50.258207ms +2024-07-05T13:35:41.841Z [ERROR] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: failed to heartbeat to: peer=d6fd6f1b-0f43-ced8-1bb3-379a603e4079 backoff time=20ms error="failed to connect to peer: d6fd6f1b-0f43-ced8-1bb3-379a603e4079" +2024-07-05T13:35:41.848Z [ERROR] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: failed to appendEntries to: peer="{Voter server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 d6fd6f1b-0f43-ced8-1bb3-379a603e4079}" error="failed to connect to peer: d6fd6f1b-0f43-ced8-1bb3-379a603e4079" +2024-07-05T13:35:41.870Z [ERROR] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: failed to heartbeat to: peer=d6fd6f1b-0f43-ced8-1bb3-379a603e4079 backoff time=25ms error="failed to connect to peer: d6fd6f1b-0f43-ced8-1bb3-379a603e4079" +2024-07-05T13:35:41.878Z [WARN] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: failed to contact: server-id=server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 time=99.372227ms +2024-07-05T13:35:41.884Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:41.77904098 +0000 UTC m=+7.774962432 (1 nodes), 144.241115ms from start of poll, 434.654727ms from cluster start. Timeout at 2024-07-05 13:35:41.884364557 +0000 UTC m=+7.880286008, 105.323576ms after stability + raft_test.go:413: [INFO] Reconnecting Node at d6fd6f1b-0f43-ced8-1bb3-379a603e4079 [Candidate] +2024-07-05T13:35:41.898Z [WARN] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: Election timeout reached, restarting election +2024-07-05T13:35:41.898Z [INFO] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: entering candidate state: node="Node at d6fd6f1b-0f43-ced8-1bb3-379a603e4079 [Candidate]" term=5 +2024-07-05T13:35:41.899Z [WARN] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: rejecting vote request since we have a leader: from=d6fd6f1b-0f43-ced8-1bb3-379a603e4079 leader=0cd44dc2-28eb-71b8-06e8-1728a102716c leader-id=server-0cd44dc2-28eb-71b8-06e8-1728a102716c +2024-07-05T13:35:41.899Z [ERROR] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: peer has newer term, stopping replication: peer="{Voter server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 d6fd6f1b-0f43-ced8-1bb3-379a603e4079}" +2024-07-05T13:35:41.899Z [WARN] server-3c146ef7-4a33-8738-718b-24adb7fd9c7e: rejecting vote request since we have a leader: from=d6fd6f1b-0f43-ced8-1bb3-379a603e4079 leader=0cd44dc2-28eb-71b8-06e8-1728a102716c leader-id=server-0cd44dc2-28eb-71b8-06e8-1728a102716c +2024-07-05T13:35:41.899Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: entering follower state: follower="Node at 0cd44dc2-28eb-71b8-06e8-1728a102716c [Follower]" leader-address= leader-id= +2024-07-05T13:35:41.900Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: aborting pipeline replication: peer="{Voter server-3c146ef7-4a33-8738-718b-24adb7fd9c7e 3c146ef7-4a33-8738-718b-24adb7fd9c7e}" +2024-07-05T13:35:41.961Z [WARN] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: Election timeout reached, restarting election +2024-07-05T13:35:41.961Z [INFO] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: entering candidate state: node="Node at d6fd6f1b-0f43-ced8-1bb3-379a603e4079 [Candidate]" term=6 +2024-07-05T13:35:41.961Z [WARN] server-3c146ef7-4a33-8738-718b-24adb7fd9c7e: rejecting vote request since we have a leader: from=d6fd6f1b-0f43-ced8-1bb3-379a603e4079 leader=0cd44dc2-28eb-71b8-06e8-1728a102716c leader-id=server-0cd44dc2-28eb-71b8-06e8-1728a102716c +2024-07-05T13:35:41.961Z [WARN] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: rejecting vote request since our last term is greater: candidate=d6fd6f1b-0f43-ced8-1bb3-379a603e4079 last-term=4 last-candidate-term=2 +2024-07-05T13:35:41.992Z [WARN] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:41.992Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: entering candidate state: node="Node at 0cd44dc2-28eb-71b8-06e8-1728a102716c [Candidate]" term=7 +2024-07-05T13:35:41.992Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: election won: term=7 tally=2 +2024-07-05T13:35:41.992Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: entering leader state: leader="Node at 0cd44dc2-28eb-71b8-06e8-1728a102716c [Leader]" +2024-07-05T13:35:41.992Z [INFO] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: entering follower state: follower="Node at d6fd6f1b-0f43-ced8-1bb3-379a603e4079 [Follower]" leader-address= leader-id= +2024-07-05T13:35:41.993Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: added peer, starting replication: peer=server-3c146ef7-4a33-8738-718b-24adb7fd9c7e +2024-07-05T13:35:41.993Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: added peer, starting replication: peer=server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 +2024-07-05T13:35:41.993Z [WARN] server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079: failed to get previous log: previous-index=5 last-index=3 error="log not found" +2024-07-05T13:35:41.995Z [WARN] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: appendEntries rejected, sending older logs: peer="{Voter server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 d6fd6f1b-0f43-ced8-1bb3-379a603e4079}" next=4 +2024-07-05T13:35:41.996Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: pipelining replication: peer="{Voter server-3c146ef7-4a33-8738-718b-24adb7fd9c7e 3c146ef7-4a33-8738-718b-24adb7fd9c7e}" +2024-07-05T13:35:41.998Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: pipelining replication: peer="{Voter server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 d6fd6f1b-0f43-ced8-1bb3-379a603e4079}" +2024-07-05T13:35:42.001Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: aborting pipeline replication: peer="{Voter server-d6fd6f1b-0f43-ced8-1bb3-379a603e4079 d6fd6f1b-0f43-ced8-1bb3-379a603e4079}" +2024-07-05T13:35:42.001Z [INFO] server-0cd44dc2-28eb-71b8-06e8-1728a102716c: aborting pipeline replication: peer="{Voter server-3c146ef7-4a33-8738-718b-24adb7fd9c7e 3c146ef7-4a33-8738-718b-24adb7fd9c7e}" +--- PASS: TestRaft_LeaderFail (0.67s) === RUN TestRaft_BehindFollower -2024-07-05T13:29:26.780Z [INFO] server-a8176a5b-797f-3570-2d2e-daa8101112a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8176a5b-797f-3570-2d2e-daa8101112a0 Address:a8176a5b-797f-3570-2d2e-daa8101112a0} {Suffrage:Voter ID:server-ef417757-cbd7-91aa-d388-4a0e5d7cb0e7 Address:ef417757-cbd7-91aa-d388-4a0e5d7cb0e7} {Suffrage:Voter ID:server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2 Address:669de2b5-d25d-e4a4-e014-d9eaf669cbc2}]" -2024-07-05T13:29:26.780Z [INFO] server-a8176a5b-797f-3570-2d2e-daa8101112a0: entering follower state: follower="Node at a8176a5b-797f-3570-2d2e-daa8101112a0 [Follower]" leader-address= leader-id= -2024-07-05T13:29:26.781Z [INFO] server-ef417757-cbd7-91aa-d388-4a0e5d7cb0e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8176a5b-797f-3570-2d2e-daa8101112a0 Address:a8176a5b-797f-3570-2d2e-daa8101112a0} {Suffrage:Voter ID:server-ef417757-cbd7-91aa-d388-4a0e5d7cb0e7 Address:ef417757-cbd7-91aa-d388-4a0e5d7cb0e7} {Suffrage:Voter ID:server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2 Address:669de2b5-d25d-e4a4-e014-d9eaf669cbc2}]" -2024-07-05T13:29:26.781Z [INFO] server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8176a5b-797f-3570-2d2e-daa8101112a0 Address:a8176a5b-797f-3570-2d2e-daa8101112a0} {Suffrage:Voter ID:server-ef417757-cbd7-91aa-d388-4a0e5d7cb0e7 Address:ef417757-cbd7-91aa-d388-4a0e5d7cb0e7} {Suffrage:Voter ID:server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2 Address:669de2b5-d25d-e4a4-e014-d9eaf669cbc2}]" -2024-07-05T13:29:26.781Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:26.781Z [INFO] server-ef417757-cbd7-91aa-d388-4a0e5d7cb0e7: entering follower state: follower="Node at ef417757-cbd7-91aa-d388-4a0e5d7cb0e7 [Follower]" leader-address= leader-id= -2024-07-05T13:29:26.781Z [INFO] server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2: entering follower state: follower="Node at 669de2b5-d25d-e4a4-e014-d9eaf669cbc2 [Follower]" leader-address= leader-id= -2024-07-05T13:29:26.841Z [WARN] server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:26.841Z [INFO] server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2: entering candidate state: node="Node at 669de2b5-d25d-e4a4-e014-d9eaf669cbc2 [Candidate]" term=2 -2024-07-05T13:29:26.841Z [INFO] server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2: election won: term=2 tally=2 -2024-07-05T13:29:26.841Z [INFO] server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2: entering leader state: leader="Node at 669de2b5-d25d-e4a4-e014-d9eaf669cbc2 [Leader]" -2024-07-05T13:29:26.841Z [INFO] server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2: added peer, starting replication: peer=server-a8176a5b-797f-3570-2d2e-daa8101112a0 -2024-07-05T13:29:26.841Z [INFO] server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2: added peer, starting replication: peer=server-ef417757-cbd7-91aa-d388-4a0e5d7cb0e7 -2024-07-05T13:29:26.842Z [INFO] server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2: pipelining replication: peer="{Voter server-ef417757-cbd7-91aa-d388-4a0e5d7cb0e7 ef417757-cbd7-91aa-d388-4a0e5d7cb0e7}" -2024-07-05T13:29:26.842Z [INFO] server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2: pipelining replication: peer="{Voter server-a8176a5b-797f-3570-2d2e-daa8101112a0 a8176a5b-797f-3570-2d2e-daa8101112a0}" -2024-07-05T13:29:26.947Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:26.842360565 +0000 UTC m=+7.612271240 (1 nodes), 60.789508ms from start of poll, 61.767314ms from cluster start. Timeout at 2024-07-05 13:29:26.947715909 +0000 UTC m=+7.717626584, 105.355344ms after stability -2024-07-05T13:29:26.947Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:29:27.053Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:29:26.947854504 +0000 UTC m=+7.717765179 (2 nodes), 2.24µs from start of poll, 167.261253ms from cluster start. Timeout at 2024-07-05 13:29:27.053417961 +0000 UTC m=+7.823328636, 105.563457ms after stability -2024-07-05T13:29:27.053Z [ERROR] server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2: failed to pipeline appendEntries: peer="{Voter server-a8176a5b-797f-3570-2d2e-daa8101112a0 a8176a5b-797f-3570-2d2e-daa8101112a0}" error="append pipeline closed" -2024-07-05T13:29:27.053Z [INFO] server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2: aborting pipeline replication: peer="{Voter server-a8176a5b-797f-3570-2d2e-daa8101112a0 a8176a5b-797f-3570-2d2e-daa8101112a0}" -2024-07-05T13:29:27.053Z [ERROR] server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2: failed to appendEntries to: peer="{Voter server-a8176a5b-797f-3570-2d2e-daa8101112a0 a8176a5b-797f-3570-2d2e-daa8101112a0}" error="failed to connect to peer: a8176a5b-797f-3570-2d2e-daa8101112a0" -2024-07-05T13:29:27.055Z [ERROR] server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2: failed to heartbeat to: peer=a8176a5b-797f-3570-2d2e-daa8101112a0 backoff time=10ms error="failed to connect to peer: a8176a5b-797f-3570-2d2e-daa8101112a0" +2024-07-05T13:35:42.013Z [INFO] server-31771b76-08b9-55e7-51c5-65e1f388c015: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31771b76-08b9-55e7-51c5-65e1f388c015 Address:31771b76-08b9-55e7-51c5-65e1f388c015} {Suffrage:Voter ID:server-fded7e24-5065-95c9-2493-3441f01b87cc Address:fded7e24-5065-95c9-2493-3441f01b87cc} {Suffrage:Voter ID:server-9bdd3e1b-ba3a-02bf-0b15-bf584d10d1d8 Address:9bdd3e1b-ba3a-02bf-0b15-bf584d10d1d8}]" +2024-07-05T13:35:42.014Z [INFO] server-31771b76-08b9-55e7-51c5-65e1f388c015: entering follower state: follower="Node at 31771b76-08b9-55e7-51c5-65e1f388c015 [Follower]" leader-address= leader-id= +2024-07-05T13:35:42.014Z [INFO] server-fded7e24-5065-95c9-2493-3441f01b87cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31771b76-08b9-55e7-51c5-65e1f388c015 Address:31771b76-08b9-55e7-51c5-65e1f388c015} {Suffrage:Voter ID:server-fded7e24-5065-95c9-2493-3441f01b87cc Address:fded7e24-5065-95c9-2493-3441f01b87cc} {Suffrage:Voter ID:server-9bdd3e1b-ba3a-02bf-0b15-bf584d10d1d8 Address:9bdd3e1b-ba3a-02bf-0b15-bf584d10d1d8}]" +2024-07-05T13:35:42.014Z [INFO] server-fded7e24-5065-95c9-2493-3441f01b87cc: entering follower state: follower="Node at fded7e24-5065-95c9-2493-3441f01b87cc [Follower]" leader-address= leader-id= +2024-07-05T13:35:42.015Z [INFO] server-9bdd3e1b-ba3a-02bf-0b15-bf584d10d1d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31771b76-08b9-55e7-51c5-65e1f388c015 Address:31771b76-08b9-55e7-51c5-65e1f388c015} {Suffrage:Voter ID:server-fded7e24-5065-95c9-2493-3441f01b87cc Address:fded7e24-5065-95c9-2493-3441f01b87cc} {Suffrage:Voter ID:server-9bdd3e1b-ba3a-02bf-0b15-bf584d10d1d8 Address:9bdd3e1b-ba3a-02bf-0b15-bf584d10d1d8}]" +2024-07-05T13:35:42.015Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:42.016Z [INFO] server-9bdd3e1b-ba3a-02bf-0b15-bf584d10d1d8: entering follower state: follower="Node at 9bdd3e1b-ba3a-02bf-0b15-bf584d10d1d8 [Follower]" leader-address= leader-id= +2024-07-05T13:35:42.089Z [WARN] server-fded7e24-5065-95c9-2493-3441f01b87cc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:42.089Z [INFO] server-fded7e24-5065-95c9-2493-3441f01b87cc: entering candidate state: node="Node at fded7e24-5065-95c9-2493-3441f01b87cc [Candidate]" term=2 +2024-07-05T13:35:42.090Z [INFO] server-fded7e24-5065-95c9-2493-3441f01b87cc: election won: term=2 tally=2 +2024-07-05T13:35:42.090Z [INFO] server-fded7e24-5065-95c9-2493-3441f01b87cc: entering leader state: leader="Node at fded7e24-5065-95c9-2493-3441f01b87cc [Leader]" +2024-07-05T13:35:42.090Z [INFO] server-fded7e24-5065-95c9-2493-3441f01b87cc: added peer, starting replication: peer=server-31771b76-08b9-55e7-51c5-65e1f388c015 +2024-07-05T13:35:42.090Z [INFO] server-fded7e24-5065-95c9-2493-3441f01b87cc: added peer, starting replication: peer=server-9bdd3e1b-ba3a-02bf-0b15-bf584d10d1d8 +2024-07-05T13:35:42.091Z [INFO] server-fded7e24-5065-95c9-2493-3441f01b87cc: pipelining replication: peer="{Voter server-31771b76-08b9-55e7-51c5-65e1f388c015 31771b76-08b9-55e7-51c5-65e1f388c015}" +2024-07-05T13:35:42.091Z [INFO] server-fded7e24-5065-95c9-2493-3441f01b87cc: pipelining replication: peer="{Voter server-9bdd3e1b-ba3a-02bf-0b15-bf584d10d1d8 9bdd3e1b-ba3a-02bf-0b15-bf584d10d1d8}" +2024-07-05T13:35:42.195Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:42.090498541 +0000 UTC m=+8.086419993 (1 nodes), 74.472515ms from start of poll, 77.634999ms from cluster start. Timeout at 2024-07-05 13:35:42.195559039 +0000 UTC m=+8.191480490, 105.060497ms after stability +2024-07-05T13:35:42.195Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:35:42.301Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:35:42.195967073 +0000 UTC m=+8.191888524 (2 nodes), 9.041µs from start of poll, 183.10353ms from cluster start. Timeout at 2024-07-05 13:35:42.301463021 +0000 UTC m=+8.297384472, 105.495948ms after stability +2024-07-05T13:35:42.302Z [ERROR] server-fded7e24-5065-95c9-2493-3441f01b87cc: failed to pipeline appendEntries: peer="{Voter server-31771b76-08b9-55e7-51c5-65e1f388c015 31771b76-08b9-55e7-51c5-65e1f388c015}" error="append pipeline closed" +2024-07-05T13:35:42.302Z [INFO] server-fded7e24-5065-95c9-2493-3441f01b87cc: aborting pipeline replication: peer="{Voter server-31771b76-08b9-55e7-51c5-65e1f388c015 31771b76-08b9-55e7-51c5-65e1f388c015}" +2024-07-05T13:35:42.302Z [ERROR] server-fded7e24-5065-95c9-2493-3441f01b87cc: failed to appendEntries to: peer="{Voter server-31771b76-08b9-55e7-51c5-65e1f388c015 31771b76-08b9-55e7-51c5-65e1f388c015}" error="failed to connect to peer: 31771b76-08b9-55e7-51c5-65e1f388c015" +2024-07-05T13:35:42.305Z [ERROR] server-fded7e24-5065-95c9-2493-3441f01b87cc: failed to heartbeat to: peer=31771b76-08b9-55e7-51c5-65e1f388c015 backoff time=10ms error="failed to connect to peer: 31771b76-08b9-55e7-51c5-65e1f388c015" +2024-07-05T13:35:42.314Z [ERROR] server-fded7e24-5065-95c9-2493-3441f01b87cc: failed to appendEntries to: peer="{Voter server-31771b76-08b9-55e7-51c5-65e1f388c015 31771b76-08b9-55e7-51c5-65e1f388c015}" error="failed to connect to peer: 31771b76-08b9-55e7-51c5-65e1f388c015" +2024-07-05T13:35:42.324Z [ERROR] server-fded7e24-5065-95c9-2493-3441f01b87cc: failed to appendEntries to: peer="{Voter server-31771b76-08b9-55e7-51c5-65e1f388c015 31771b76-08b9-55e7-51c5-65e1f388c015}" error="failed to connect to peer: 31771b76-08b9-55e7-51c5-65e1f388c015" +2024-07-05T13:35:42.325Z [ERROR] server-fded7e24-5065-95c9-2493-3441f01b87cc: failed to heartbeat to: peer=31771b76-08b9-55e7-51c5-65e1f388c015 backoff time=10ms error="failed to connect to peer: 31771b76-08b9-55e7-51c5-65e1f388c015" raft_test.go:459: [INFO] Finished apply without behind follower -2024-07-05T13:29:27.065Z [INFO] server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2: pipelining replication: peer="{Voter server-a8176a5b-797f-3570-2d2e-daa8101112a0 a8176a5b-797f-3570-2d2e-daa8101112a0}" -2024-07-05T13:29:27.080Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:27.185Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:27.080350456 +0000 UTC m=+7.850261131 (1 nodes), 3.2µs from start of poll, 299.757205ms from cluster start. Timeout at 2024-07-05 13:29:27.185372092 +0000 UTC m=+7.955282767, 105.021636ms after stability -2024-07-05T13:29:27.185Z [INFO] server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2: aborting pipeline replication: peer="{Voter server-ef417757-cbd7-91aa-d388-4a0e5d7cb0e7 ef417757-cbd7-91aa-d388-4a0e5d7cb0e7}" -2024-07-05T13:29:27.185Z [INFO] server-669de2b5-d25d-e4a4-e014-d9eaf669cbc2: aborting pipeline replication: peer="{Voter server-a8176a5b-797f-3570-2d2e-daa8101112a0 a8176a5b-797f-3570-2d2e-daa8101112a0}" ---- PASS: TestRaft_BehindFollower (0.41s) +2024-07-05T13:35:42.347Z [INFO] server-fded7e24-5065-95c9-2493-3441f01b87cc: pipelining replication: peer="{Voter server-31771b76-08b9-55e7-51c5-65e1f388c015 31771b76-08b9-55e7-51c5-65e1f388c015}" +2024-07-05T13:35:42.353Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:42.459Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:42.353813023 +0000 UTC m=+8.349734474 (1 nodes), 6.124µs from start of poll, 340.94948ms from cluster start. Timeout at 2024-07-05 13:35:42.459563883 +0000 UTC m=+8.455485334, 105.75086ms after stability +2024-07-05T13:35:42.460Z [INFO] server-fded7e24-5065-95c9-2493-3441f01b87cc: aborting pipeline replication: peer="{Voter server-31771b76-08b9-55e7-51c5-65e1f388c015 31771b76-08b9-55e7-51c5-65e1f388c015}" +2024-07-05T13:35:42.460Z [INFO] server-fded7e24-5065-95c9-2493-3441f01b87cc: aborting pipeline replication: peer="{Voter server-9bdd3e1b-ba3a-02bf-0b15-bf584d10d1d8 9bdd3e1b-ba3a-02bf-0b15-bf584d10d1d8}" +--- PASS: TestRaft_BehindFollower (0.46s) === RUN TestRaft_ApplyNonLeader -2024-07-05T13:29:27.187Z [INFO] server-cafe27b8-c2fd-17eb-ce70-2496e664eb9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cafe27b8-c2fd-17eb-ce70-2496e664eb9f Address:cafe27b8-c2fd-17eb-ce70-2496e664eb9f} {Suffrage:Voter ID:server-6ec1e79e-6c32-3938-f30e-bf8ad87fee5f Address:6ec1e79e-6c32-3938-f30e-bf8ad87fee5f} {Suffrage:Voter ID:server-840e4914-cfb2-5291-6254-e30a5bf31699 Address:840e4914-cfb2-5291-6254-e30a5bf31699}]" -2024-07-05T13:29:27.188Z [INFO] server-cafe27b8-c2fd-17eb-ce70-2496e664eb9f: entering follower state: follower="Node at cafe27b8-c2fd-17eb-ce70-2496e664eb9f [Follower]" leader-address= leader-id= -2024-07-05T13:29:27.188Z [INFO] server-6ec1e79e-6c32-3938-f30e-bf8ad87fee5f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cafe27b8-c2fd-17eb-ce70-2496e664eb9f Address:cafe27b8-c2fd-17eb-ce70-2496e664eb9f} {Suffrage:Voter ID:server-6ec1e79e-6c32-3938-f30e-bf8ad87fee5f Address:6ec1e79e-6c32-3938-f30e-bf8ad87fee5f} {Suffrage:Voter ID:server-840e4914-cfb2-5291-6254-e30a5bf31699 Address:840e4914-cfb2-5291-6254-e30a5bf31699}]" -2024-07-05T13:29:27.189Z [INFO] server-840e4914-cfb2-5291-6254-e30a5bf31699: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cafe27b8-c2fd-17eb-ce70-2496e664eb9f Address:cafe27b8-c2fd-17eb-ce70-2496e664eb9f} {Suffrage:Voter ID:server-6ec1e79e-6c32-3938-f30e-bf8ad87fee5f Address:6ec1e79e-6c32-3938-f30e-bf8ad87fee5f} {Suffrage:Voter ID:server-840e4914-cfb2-5291-6254-e30a5bf31699 Address:840e4914-cfb2-5291-6254-e30a5bf31699}]" -2024-07-05T13:29:27.189Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:27.189Z [INFO] server-6ec1e79e-6c32-3938-f30e-bf8ad87fee5f: entering follower state: follower="Node at 6ec1e79e-6c32-3938-f30e-bf8ad87fee5f [Follower]" leader-address= leader-id= -2024-07-05T13:29:27.189Z [INFO] server-840e4914-cfb2-5291-6254-e30a5bf31699: entering follower state: follower="Node at 840e4914-cfb2-5291-6254-e30a5bf31699 [Follower]" leader-address= leader-id= -2024-07-05T13:29:27.260Z [WARN] server-840e4914-cfb2-5291-6254-e30a5bf31699: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:27.260Z [INFO] server-840e4914-cfb2-5291-6254-e30a5bf31699: entering candidate state: node="Node at 840e4914-cfb2-5291-6254-e30a5bf31699 [Candidate]" term=2 -2024-07-05T13:29:27.260Z [INFO] server-840e4914-cfb2-5291-6254-e30a5bf31699: election won: term=2 tally=2 -2024-07-05T13:29:27.260Z [INFO] server-840e4914-cfb2-5291-6254-e30a5bf31699: entering leader state: leader="Node at 840e4914-cfb2-5291-6254-e30a5bf31699 [Leader]" -2024-07-05T13:29:27.260Z [INFO] server-840e4914-cfb2-5291-6254-e30a5bf31699: added peer, starting replication: peer=server-cafe27b8-c2fd-17eb-ce70-2496e664eb9f -2024-07-05T13:29:27.260Z [INFO] server-840e4914-cfb2-5291-6254-e30a5bf31699: added peer, starting replication: peer=server-6ec1e79e-6c32-3938-f30e-bf8ad87fee5f -2024-07-05T13:29:27.260Z [INFO] server-840e4914-cfb2-5291-6254-e30a5bf31699: pipelining replication: peer="{Voter server-6ec1e79e-6c32-3938-f30e-bf8ad87fee5f 6ec1e79e-6c32-3938-f30e-bf8ad87fee5f}" -2024-07-05T13:29:27.260Z [INFO] server-840e4914-cfb2-5291-6254-e30a5bf31699: pipelining replication: peer="{Voter server-cafe27b8-c2fd-17eb-ce70-2496e664eb9f cafe27b8-c2fd-17eb-ce70-2496e664eb9f}" -2024-07-05T13:29:27.367Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:27.261101395 +0000 UTC m=+8.031012070 (1 nodes), 71.929957ms from start of poll, 73.383746ms from cluster start. Timeout at 2024-07-05 13:29:27.36698884 +0000 UTC m=+8.136899515, 105.887445ms after stability -2024-07-05T13:29:27.367Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:29:27.472Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:29:27.367127755 +0000 UTC m=+8.137038430 (2 nodes), 1.959µs from start of poll, 179.410106ms from cluster start. Timeout at 2024-07-05 13:29:27.472512698 +0000 UTC m=+8.242423373, 105.384943ms after stability -2024-07-05T13:29:27.472Z [INFO] server-840e4914-cfb2-5291-6254-e30a5bf31699: aborting pipeline replication: peer="{Voter server-6ec1e79e-6c32-3938-f30e-bf8ad87fee5f 6ec1e79e-6c32-3938-f30e-bf8ad87fee5f}" -2024-07-05T13:29:27.472Z [INFO] server-840e4914-cfb2-5291-6254-e30a5bf31699: aborting pipeline replication: peer="{Voter server-cafe27b8-c2fd-17eb-ce70-2496e664eb9f cafe27b8-c2fd-17eb-ce70-2496e664eb9f}" +2024-07-05T13:35:42.471Z [INFO] server-ba46ec28-8cfd-1897-01cd-ebd7ef371b38: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba46ec28-8cfd-1897-01cd-ebd7ef371b38 Address:ba46ec28-8cfd-1897-01cd-ebd7ef371b38} {Suffrage:Voter ID:server-eed156f0-a0e9-124b-077d-7ebc224bb36c Address:eed156f0-a0e9-124b-077d-7ebc224bb36c} {Suffrage:Voter ID:server-cf322c46-9e86-8a17-6397-aeee28d3cc01 Address:cf322c46-9e86-8a17-6397-aeee28d3cc01}]" +2024-07-05T13:35:42.471Z [INFO] server-ba46ec28-8cfd-1897-01cd-ebd7ef371b38: entering follower state: follower="Node at ba46ec28-8cfd-1897-01cd-ebd7ef371b38 [Follower]" leader-address= leader-id= +2024-07-05T13:35:42.472Z [INFO] server-eed156f0-a0e9-124b-077d-7ebc224bb36c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba46ec28-8cfd-1897-01cd-ebd7ef371b38 Address:ba46ec28-8cfd-1897-01cd-ebd7ef371b38} {Suffrage:Voter ID:server-eed156f0-a0e9-124b-077d-7ebc224bb36c Address:eed156f0-a0e9-124b-077d-7ebc224bb36c} {Suffrage:Voter ID:server-cf322c46-9e86-8a17-6397-aeee28d3cc01 Address:cf322c46-9e86-8a17-6397-aeee28d3cc01}]" +2024-07-05T13:35:42.472Z [INFO] server-eed156f0-a0e9-124b-077d-7ebc224bb36c: entering follower state: follower="Node at eed156f0-a0e9-124b-077d-7ebc224bb36c [Follower]" leader-address= leader-id= +2024-07-05T13:35:42.473Z [INFO] server-cf322c46-9e86-8a17-6397-aeee28d3cc01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba46ec28-8cfd-1897-01cd-ebd7ef371b38 Address:ba46ec28-8cfd-1897-01cd-ebd7ef371b38} {Suffrage:Voter ID:server-eed156f0-a0e9-124b-077d-7ebc224bb36c Address:eed156f0-a0e9-124b-077d-7ebc224bb36c} {Suffrage:Voter ID:server-cf322c46-9e86-8a17-6397-aeee28d3cc01 Address:cf322c46-9e86-8a17-6397-aeee28d3cc01}]" +2024-07-05T13:35:42.473Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:42.473Z [INFO] server-cf322c46-9e86-8a17-6397-aeee28d3cc01: entering follower state: follower="Node at cf322c46-9e86-8a17-6397-aeee28d3cc01 [Follower]" leader-address= leader-id= +2024-07-05T13:35:42.534Z [WARN] server-ba46ec28-8cfd-1897-01cd-ebd7ef371b38: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:42.534Z [INFO] server-ba46ec28-8cfd-1897-01cd-ebd7ef371b38: entering candidate state: node="Node at ba46ec28-8cfd-1897-01cd-ebd7ef371b38 [Candidate]" term=2 +2024-07-05T13:35:42.534Z [INFO] server-ba46ec28-8cfd-1897-01cd-ebd7ef371b38: election won: term=2 tally=2 +2024-07-05T13:35:42.534Z [INFO] server-ba46ec28-8cfd-1897-01cd-ebd7ef371b38: entering leader state: leader="Node at ba46ec28-8cfd-1897-01cd-ebd7ef371b38 [Leader]" +2024-07-05T13:35:42.535Z [INFO] server-ba46ec28-8cfd-1897-01cd-ebd7ef371b38: added peer, starting replication: peer=server-eed156f0-a0e9-124b-077d-7ebc224bb36c +2024-07-05T13:35:42.535Z [INFO] server-ba46ec28-8cfd-1897-01cd-ebd7ef371b38: added peer, starting replication: peer=server-cf322c46-9e86-8a17-6397-aeee28d3cc01 +2024-07-05T13:35:42.536Z [INFO] server-ba46ec28-8cfd-1897-01cd-ebd7ef371b38: pipelining replication: peer="{Voter server-eed156f0-a0e9-124b-077d-7ebc224bb36c eed156f0-a0e9-124b-077d-7ebc224bb36c}" +2024-07-05T13:35:42.536Z [INFO] server-ba46ec28-8cfd-1897-01cd-ebd7ef371b38: pipelining replication: peer="{Voter server-cf322c46-9e86-8a17-6397-aeee28d3cc01 cf322c46-9e86-8a17-6397-aeee28d3cc01}" +2024-07-05T13:35:42.640Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:42.535110294 +0000 UTC m=+8.531031746 (1 nodes), 61.67529ms from start of poll, 64.533279ms from cluster start. Timeout at 2024-07-05 13:35:42.640146876 +0000 UTC m=+8.636068036, 105.03629ms after stability +2024-07-05T13:35:42.640Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:35:42.746Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:35:42.640509994 +0000 UTC m=+8.636431446 (2 nodes), 8.167µs from start of poll, 169.932979ms from cluster start. Timeout at 2024-07-05 13:35:42.746033358 +0000 UTC m=+8.741954810, 105.523364ms after stability +2024-07-05T13:35:42.746Z [INFO] server-ba46ec28-8cfd-1897-01cd-ebd7ef371b38: aborting pipeline replication: peer="{Voter server-eed156f0-a0e9-124b-077d-7ebc224bb36c eed156f0-a0e9-124b-077d-7ebc224bb36c}" +2024-07-05T13:35:42.746Z [INFO] server-ba46ec28-8cfd-1897-01cd-ebd7ef371b38: aborting pipeline replication: peer="{Voter server-cf322c46-9e86-8a17-6397-aeee28d3cc01 cf322c46-9e86-8a17-6397-aeee28d3cc01}" --- PASS: TestRaft_ApplyNonLeader (0.29s) === RUN TestRaft_ApplyConcurrent -2024-07-05T13:29:27.474Z [INFO] server-459c354f-2db3-62af-6d02-72de8c67934b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-459c354f-2db3-62af-6d02-72de8c67934b Address:459c354f-2db3-62af-6d02-72de8c67934b} {Suffrage:Voter ID:server-43bbf28c-83b2-d15b-3062-0d96073f5c6c Address:43bbf28c-83b2-d15b-3062-0d96073f5c6c} {Suffrage:Voter ID:server-51d54be6-c21f-84dc-0e7d-42205bd74b2f Address:51d54be6-c21f-84dc-0e7d-42205bd74b2f}]" -2024-07-05T13:29:27.475Z [INFO] server-43bbf28c-83b2-d15b-3062-0d96073f5c6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-459c354f-2db3-62af-6d02-72de8c67934b Address:459c354f-2db3-62af-6d02-72de8c67934b} {Suffrage:Voter ID:server-43bbf28c-83b2-d15b-3062-0d96073f5c6c Address:43bbf28c-83b2-d15b-3062-0d96073f5c6c} {Suffrage:Voter ID:server-51d54be6-c21f-84dc-0e7d-42205bd74b2f Address:51d54be6-c21f-84dc-0e7d-42205bd74b2f}]" -2024-07-05T13:29:27.475Z [INFO] server-51d54be6-c21f-84dc-0e7d-42205bd74b2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-459c354f-2db3-62af-6d02-72de8c67934b Address:459c354f-2db3-62af-6d02-72de8c67934b} {Suffrage:Voter ID:server-43bbf28c-83b2-d15b-3062-0d96073f5c6c Address:43bbf28c-83b2-d15b-3062-0d96073f5c6c} {Suffrage:Voter ID:server-51d54be6-c21f-84dc-0e7d-42205bd74b2f Address:51d54be6-c21f-84dc-0e7d-42205bd74b2f}]" -2024-07-05T13:29:27.475Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:27.475Z [INFO] server-459c354f-2db3-62af-6d02-72de8c67934b: entering follower state: follower="Node at 459c354f-2db3-62af-6d02-72de8c67934b [Follower]" leader-address= leader-id= -2024-07-05T13:29:27.475Z [INFO] server-43bbf28c-83b2-d15b-3062-0d96073f5c6c: entering follower state: follower="Node at 43bbf28c-83b2-d15b-3062-0d96073f5c6c [Follower]" leader-address= leader-id= -2024-07-05T13:29:27.475Z [INFO] server-51d54be6-c21f-84dc-0e7d-42205bd74b2f: entering follower state: follower="Node at 51d54be6-c21f-84dc-0e7d-42205bd74b2f [Follower]" leader-address= leader-id= -2024-07-05T13:29:27.576Z [WARN] server-51d54be6-c21f-84dc-0e7d-42205bd74b2f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:27.576Z [INFO] server-51d54be6-c21f-84dc-0e7d-42205bd74b2f: entering candidate state: node="Node at 51d54be6-c21f-84dc-0e7d-42205bd74b2f [Candidate]" term=2 -2024-07-05T13:29:27.576Z [INFO] server-51d54be6-c21f-84dc-0e7d-42205bd74b2f: election won: term=2 tally=2 -2024-07-05T13:29:27.576Z [INFO] server-51d54be6-c21f-84dc-0e7d-42205bd74b2f: entering leader state: leader="Node at 51d54be6-c21f-84dc-0e7d-42205bd74b2f [Leader]" -2024-07-05T13:29:27.576Z [INFO] server-51d54be6-c21f-84dc-0e7d-42205bd74b2f: added peer, starting replication: peer=server-459c354f-2db3-62af-6d02-72de8c67934b -2024-07-05T13:29:27.576Z [INFO] server-51d54be6-c21f-84dc-0e7d-42205bd74b2f: added peer, starting replication: peer=server-43bbf28c-83b2-d15b-3062-0d96073f5c6c -2024-07-05T13:29:27.576Z [INFO] server-51d54be6-c21f-84dc-0e7d-42205bd74b2f: pipelining replication: peer="{Voter server-43bbf28c-83b2-d15b-3062-0d96073f5c6c 43bbf28c-83b2-d15b-3062-0d96073f5c6c}" -2024-07-05T13:29:27.577Z [INFO] server-51d54be6-c21f-84dc-0e7d-42205bd74b2f: pipelining replication: peer="{Voter server-459c354f-2db3-62af-6d02-72de8c67934b 459c354f-2db3-62af-6d02-72de8c67934b}" -2024-07-05T13:29:27.782Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:27.577103989 +0000 UTC m=+8.347014664 (1 nodes), 101.689192ms from start of poll, 102.350769ms from cluster start. Timeout at 2024-07-05 13:29:27.782348029 +0000 UTC m=+8.552258704, 205.24404ms after stability -2024-07-05T13:29:27.800Z [INFO] server-51d54be6-c21f-84dc-0e7d-42205bd74b2f: aborting pipeline replication: peer="{Voter server-459c354f-2db3-62af-6d02-72de8c67934b 459c354f-2db3-62af-6d02-72de8c67934b}" -2024-07-05T13:29:27.800Z [INFO] server-51d54be6-c21f-84dc-0e7d-42205bd74b2f: aborting pipeline replication: peer="{Voter server-43bbf28c-83b2-d15b-3062-0d96073f5c6c 43bbf28c-83b2-d15b-3062-0d96073f5c6c}" ---- PASS: TestRaft_ApplyConcurrent (0.33s) +2024-07-05T13:35:42.758Z [INFO] server-469c715b-7ec5-b16e-f1ef-065b037c570a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-469c715b-7ec5-b16e-f1ef-065b037c570a Address:469c715b-7ec5-b16e-f1ef-065b037c570a} {Suffrage:Voter ID:server-172b91c2-3106-e580-d14a-9e51b3ad1651 Address:172b91c2-3106-e580-d14a-9e51b3ad1651} {Suffrage:Voter ID:server-33ba46e5-65d6-a3b9-c775-accc6296aabe Address:33ba46e5-65d6-a3b9-c775-accc6296aabe}]" +2024-07-05T13:35:42.758Z [INFO] server-469c715b-7ec5-b16e-f1ef-065b037c570a: entering follower state: follower="Node at 469c715b-7ec5-b16e-f1ef-065b037c570a [Follower]" leader-address= leader-id= +2024-07-05T13:35:42.759Z [INFO] server-172b91c2-3106-e580-d14a-9e51b3ad1651: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-469c715b-7ec5-b16e-f1ef-065b037c570a Address:469c715b-7ec5-b16e-f1ef-065b037c570a} {Suffrage:Voter ID:server-172b91c2-3106-e580-d14a-9e51b3ad1651 Address:172b91c2-3106-e580-d14a-9e51b3ad1651} {Suffrage:Voter ID:server-33ba46e5-65d6-a3b9-c775-accc6296aabe Address:33ba46e5-65d6-a3b9-c775-accc6296aabe}]" +2024-07-05T13:35:42.759Z [INFO] server-172b91c2-3106-e580-d14a-9e51b3ad1651: entering follower state: follower="Node at 172b91c2-3106-e580-d14a-9e51b3ad1651 [Follower]" leader-address= leader-id= +2024-07-05T13:35:42.759Z [INFO] server-33ba46e5-65d6-a3b9-c775-accc6296aabe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-469c715b-7ec5-b16e-f1ef-065b037c570a Address:469c715b-7ec5-b16e-f1ef-065b037c570a} {Suffrage:Voter ID:server-172b91c2-3106-e580-d14a-9e51b3ad1651 Address:172b91c2-3106-e580-d14a-9e51b3ad1651} {Suffrage:Voter ID:server-33ba46e5-65d6-a3b9-c775-accc6296aabe Address:33ba46e5-65d6-a3b9-c775-accc6296aabe}]" +2024-07-05T13:35:42.760Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:42.760Z [INFO] server-33ba46e5-65d6-a3b9-c775-accc6296aabe: entering follower state: follower="Node at 33ba46e5-65d6-a3b9-c775-accc6296aabe [Follower]" leader-address= leader-id= +2024-07-05T13:35:42.895Z [WARN] server-33ba46e5-65d6-a3b9-c775-accc6296aabe: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:42.896Z [INFO] server-33ba46e5-65d6-a3b9-c775-accc6296aabe: entering candidate state: node="Node at 33ba46e5-65d6-a3b9-c775-accc6296aabe [Candidate]" term=2 +2024-07-05T13:35:42.896Z [INFO] server-33ba46e5-65d6-a3b9-c775-accc6296aabe: election won: term=2 tally=2 +2024-07-05T13:35:42.896Z [INFO] server-33ba46e5-65d6-a3b9-c775-accc6296aabe: entering leader state: leader="Node at 33ba46e5-65d6-a3b9-c775-accc6296aabe [Leader]" +2024-07-05T13:35:42.896Z [INFO] server-33ba46e5-65d6-a3b9-c775-accc6296aabe: added peer, starting replication: peer=server-469c715b-7ec5-b16e-f1ef-065b037c570a +2024-07-05T13:35:42.896Z [INFO] server-33ba46e5-65d6-a3b9-c775-accc6296aabe: added peer, starting replication: peer=server-172b91c2-3106-e580-d14a-9e51b3ad1651 +2024-07-05T13:35:42.898Z [INFO] server-33ba46e5-65d6-a3b9-c775-accc6296aabe: pipelining replication: peer="{Voter server-469c715b-7ec5-b16e-f1ef-065b037c570a 469c715b-7ec5-b16e-f1ef-065b037c570a}" +2024-07-05T13:35:42.898Z [INFO] server-33ba46e5-65d6-a3b9-c775-accc6296aabe: pipelining replication: peer="{Voter server-172b91c2-3106-e580-d14a-9e51b3ad1651 172b91c2-3106-e580-d14a-9e51b3ad1651}" +2024-07-05T13:35:43.103Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:42.897542965 +0000 UTC m=+8.893464417 (1 nodes), 137.304242ms from start of poll, 140.329061ms from cluster start. Timeout at 2024-07-05 13:35:43.103345914 +0000 UTC m=+9.099267366, 205.802949ms after stability +2024-07-05T13:35:43.126Z [INFO] server-33ba46e5-65d6-a3b9-c775-accc6296aabe: aborting pipeline replication: peer="{Voter server-172b91c2-3106-e580-d14a-9e51b3ad1651 172b91c2-3106-e580-d14a-9e51b3ad1651}" +2024-07-05T13:35:43.126Z [INFO] server-33ba46e5-65d6-a3b9-c775-accc6296aabe: aborting pipeline replication: peer="{Voter server-469c715b-7ec5-b16e-f1ef-065b037c570a 469c715b-7ec5-b16e-f1ef-065b037c570a}" +--- PASS: TestRaft_ApplyConcurrent (0.38s) === RUN TestRaft_ApplyConcurrent_Timeout -2024-07-05T13:29:27.802Z [INFO] server-aa081e66-b156-8ca2-200e-6c49e95b6def: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa081e66-b156-8ca2-200e-6c49e95b6def Address:aa081e66-b156-8ca2-200e-6c49e95b6def}]" -2024-07-05T13:29:27.802Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:27.802Z [INFO] server-aa081e66-b156-8ca2-200e-6c49e95b6def: entering follower state: follower="Node at aa081e66-b156-8ca2-200e-6c49e95b6def [Follower]" leader-address= leader-id= -2024-07-05T13:29:28.002Z [WARN] server-aa081e66-b156-8ca2-200e-6c49e95b6def: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:28.002Z [INFO] server-aa081e66-b156-8ca2-200e-6c49e95b6def: entering candidate state: node="Node at aa081e66-b156-8ca2-200e-6c49e95b6def [Candidate]" term=2 -2024-07-05T13:29:28.002Z [INFO] server-aa081e66-b156-8ca2-200e-6c49e95b6def: election won: term=2 tally=1 -2024-07-05T13:29:28.002Z [INFO] server-aa081e66-b156-8ca2-200e-6c49e95b6def: entering leader state: leader="Node at aa081e66-b156-8ca2-200e-6c49e95b6def [Leader]" -2024-07-05T13:29:28.204Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:28.00296497 +0000 UTC m=+8.772875645 (1 nodes), 200.646961ms from start of poll, 200.946271ms from cluster start. Timeout at 2024-07-05 13:29:28.204188951 +0000 UTC m=+8.974099626, 201.223981ms after stability ---- PASS: TestRaft_ApplyConcurrent_Timeout (0.41s) +2024-07-05T13:35:43.134Z [INFO] server-4e290725-4419-0f42-ef60-113f93fbf73b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e290725-4419-0f42-ef60-113f93fbf73b Address:4e290725-4419-0f42-ef60-113f93fbf73b}]" +2024-07-05T13:35:43.134Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:43.134Z [INFO] server-4e290725-4419-0f42-ef60-113f93fbf73b: entering follower state: follower="Node at 4e290725-4419-0f42-ef60-113f93fbf73b [Follower]" leader-address= leader-id= +2024-07-05T13:35:43.283Z [WARN] server-4e290725-4419-0f42-ef60-113f93fbf73b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:43.283Z [INFO] server-4e290725-4419-0f42-ef60-113f93fbf73b: entering candidate state: node="Node at 4e290725-4419-0f42-ef60-113f93fbf73b [Candidate]" term=2 +2024-07-05T13:35:43.283Z [INFO] server-4e290725-4419-0f42-ef60-113f93fbf73b: election won: term=2 tally=1 +2024-07-05T13:35:43.283Z [INFO] server-4e290725-4419-0f42-ef60-113f93fbf73b: entering leader state: leader="Node at 4e290725-4419-0f42-ef60-113f93fbf73b [Leader]" +2024-07-05T13:35:43.486Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:43.283989573 +0000 UTC m=+9.279911316 (1 nodes), 149.520768ms from start of poll, 150.569291ms from cluster start. Timeout at 2024-07-05 13:35:43.485885427 +0000 UTC m=+9.481806878, 201.895562ms after stability +--- PASS: TestRaft_ApplyConcurrent_Timeout (0.36s) === RUN TestRaft_JoinNode -2024-07-05T13:29:28.210Z [INFO] server-6cab5a79-4fd7-3d34-4189-9f30a4398461: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6cab5a79-4fd7-3d34-4189-9f30a4398461 Address:6cab5a79-4fd7-3d34-4189-9f30a4398461} {Suffrage:Voter ID:server-9969f93a-63a0-4b1a-4766-bec1e784872f Address:9969f93a-63a0-4b1a-4766-bec1e784872f}]" -2024-07-05T13:29:28.210Z [INFO] server-9969f93a-63a0-4b1a-4766-bec1e784872f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6cab5a79-4fd7-3d34-4189-9f30a4398461 Address:6cab5a79-4fd7-3d34-4189-9f30a4398461} {Suffrage:Voter ID:server-9969f93a-63a0-4b1a-4766-bec1e784872f Address:9969f93a-63a0-4b1a-4766-bec1e784872f}]" -2024-07-05T13:29:28.211Z [INFO] server-befd5966-4c96-c94c-4397-d834f376ed21: initial configuration: index=0 servers=[] -2024-07-05T13:29:28.211Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:28.211Z [INFO] server-6cab5a79-4fd7-3d34-4189-9f30a4398461: entering follower state: follower="Node at 6cab5a79-4fd7-3d34-4189-9f30a4398461 [Follower]" leader-address= leader-id= -2024-07-05T13:29:28.211Z [INFO] server-9969f93a-63a0-4b1a-4766-bec1e784872f: entering follower state: follower="Node at 9969f93a-63a0-4b1a-4766-bec1e784872f [Follower]" leader-address= leader-id= -2024-07-05T13:29:28.211Z [INFO] server-befd5966-4c96-c94c-4397-d834f376ed21: entering follower state: follower="Node at befd5966-4c96-c94c-4397-d834f376ed21 [Follower]" leader-address= leader-id= -2024-07-05T13:29:28.272Z [WARN] server-6cab5a79-4fd7-3d34-4189-9f30a4398461: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:28.272Z [INFO] server-6cab5a79-4fd7-3d34-4189-9f30a4398461: entering candidate state: node="Node at 6cab5a79-4fd7-3d34-4189-9f30a4398461 [Candidate]" term=2 -2024-07-05T13:29:28.272Z [INFO] server-6cab5a79-4fd7-3d34-4189-9f30a4398461: election won: term=2 tally=2 -2024-07-05T13:29:28.272Z [INFO] server-6cab5a79-4fd7-3d34-4189-9f30a4398461: entering leader state: leader="Node at 6cab5a79-4fd7-3d34-4189-9f30a4398461 [Leader]" -2024-07-05T13:29:28.272Z [INFO] server-6cab5a79-4fd7-3d34-4189-9f30a4398461: added peer, starting replication: peer=server-9969f93a-63a0-4b1a-4766-bec1e784872f -2024-07-05T13:29:28.272Z [INFO] server-6cab5a79-4fd7-3d34-4189-9f30a4398461: pipelining replication: peer="{Voter server-9969f93a-63a0-4b1a-4766-bec1e784872f 9969f93a-63a0-4b1a-4766-bec1e784872f}" -2024-07-05T13:29:28.301Z [WARN] server-befd5966-4c96-c94c-4397-d834f376ed21: no known peers, aborting election -2024-07-05T13:29:28.378Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:28.272566952 +0000 UTC m=+9.042477667 (1 nodes), 61.481883ms from start of poll, 62.408771ms from cluster start. Timeout at 2024-07-05 13:29:28.378142248 +0000 UTC m=+9.148052963, 105.575296ms after stability -2024-07-05T13:29:28.378Z [INFO] server-6cab5a79-4fd7-3d34-4189-9f30a4398461: updating configuration: command=AddVoter server-id=server-befd5966-4c96-c94c-4397-d834f376ed21 server-addr=befd5966-4c96-c94c-4397-d834f376ed21 servers="[{Suffrage:Voter ID:server-6cab5a79-4fd7-3d34-4189-9f30a4398461 Address:6cab5a79-4fd7-3d34-4189-9f30a4398461} {Suffrage:Voter ID:server-9969f93a-63a0-4b1a-4766-bec1e784872f Address:9969f93a-63a0-4b1a-4766-bec1e784872f} {Suffrage:Voter ID:server-befd5966-4c96-c94c-4397-d834f376ed21 Address:befd5966-4c96-c94c-4397-d834f376ed21}]" -2024-07-05T13:29:28.378Z [INFO] server-6cab5a79-4fd7-3d34-4189-9f30a4398461: added peer, starting replication: peer=server-befd5966-4c96-c94c-4397-d834f376ed21 -2024-07-05T13:29:28.378Z [WARN] server-befd5966-4c96-c94c-4397-d834f376ed21: failed to get previous log: previous-index=3 last-index=0 error="log not found" -2024-07-05T13:29:28.378Z [WARN] server-6cab5a79-4fd7-3d34-4189-9f30a4398461: appendEntries rejected, sending older logs: peer="{Voter server-befd5966-4c96-c94c-4397-d834f376ed21 befd5966-4c96-c94c-4397-d834f376ed21}" next=1 -2024-07-05T13:29:28.378Z [INFO] server-6cab5a79-4fd7-3d34-4189-9f30a4398461: pipelining replication: peer="{Voter server-befd5966-4c96-c94c-4397-d834f376ed21 befd5966-4c96-c94c-4397-d834f376ed21}" -2024-07-05T13:29:28.378Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:28.484Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:28.379007898 +0000 UTC m=+9.148918573 (1 nodes), 1.36µs from start of poll, 168.849677ms from cluster start. Timeout at 2024-07-05 13:29:28.484202408 +0000 UTC m=+9.254113083, 105.19451ms after stability -2024-07-05T13:29:28.484Z [INFO] server-6cab5a79-4fd7-3d34-4189-9f30a4398461: aborting pipeline replication: peer="{Voter server-befd5966-4c96-c94c-4397-d834f376ed21 befd5966-4c96-c94c-4397-d834f376ed21}" -2024-07-05T13:29:28.484Z [INFO] server-6cab5a79-4fd7-3d34-4189-9f30a4398461: aborting pipeline replication: peer="{Voter server-9969f93a-63a0-4b1a-4766-bec1e784872f 9969f93a-63a0-4b1a-4766-bec1e784872f}" ---- PASS: TestRaft_JoinNode (0.28s) +2024-07-05T13:35:43.493Z [INFO] server-cb9766e1-4150-1a76-984d-c97837df7b7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb9766e1-4150-1a76-984d-c97837df7b7a Address:cb9766e1-4150-1a76-984d-c97837df7b7a} {Suffrage:Voter ID:server-11989612-ef93-a440-3fd6-80d2ff507dc1 Address:11989612-ef93-a440-3fd6-80d2ff507dc1}]" +2024-07-05T13:35:43.493Z [INFO] server-cb9766e1-4150-1a76-984d-c97837df7b7a: entering follower state: follower="Node at cb9766e1-4150-1a76-984d-c97837df7b7a [Follower]" leader-address= leader-id= +2024-07-05T13:35:43.494Z [INFO] server-11989612-ef93-a440-3fd6-80d2ff507dc1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb9766e1-4150-1a76-984d-c97837df7b7a Address:cb9766e1-4150-1a76-984d-c97837df7b7a} {Suffrage:Voter ID:server-11989612-ef93-a440-3fd6-80d2ff507dc1 Address:11989612-ef93-a440-3fd6-80d2ff507dc1}]" +2024-07-05T13:35:43.494Z [INFO] server-11989612-ef93-a440-3fd6-80d2ff507dc1: entering follower state: follower="Node at 11989612-ef93-a440-3fd6-80d2ff507dc1 [Follower]" leader-address= leader-id= +2024-07-05T13:35:43.496Z [INFO] server-f347f5f9-680d-e3ad-685e-073f6bce0b45: initial configuration: index=0 servers=[] +2024-07-05T13:35:43.496Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:43.496Z [INFO] server-f347f5f9-680d-e3ad-685e-073f6bce0b45: entering follower state: follower="Node at f347f5f9-680d-e3ad-685e-073f6bce0b45 [Follower]" leader-address= leader-id= +2024-07-05T13:35:43.551Z [WARN] server-f347f5f9-680d-e3ad-685e-073f6bce0b45: no known peers, aborting election +2024-07-05T13:35:43.557Z [WARN] server-11989612-ef93-a440-3fd6-80d2ff507dc1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:43.557Z [INFO] server-11989612-ef93-a440-3fd6-80d2ff507dc1: entering candidate state: node="Node at 11989612-ef93-a440-3fd6-80d2ff507dc1 [Candidate]" term=2 +2024-07-05T13:35:43.558Z [INFO] server-11989612-ef93-a440-3fd6-80d2ff507dc1: election won: term=2 tally=2 +2024-07-05T13:35:43.558Z [INFO] server-11989612-ef93-a440-3fd6-80d2ff507dc1: entering leader state: leader="Node at 11989612-ef93-a440-3fd6-80d2ff507dc1 [Leader]" +2024-07-05T13:35:43.558Z [INFO] server-11989612-ef93-a440-3fd6-80d2ff507dc1: added peer, starting replication: peer=server-cb9766e1-4150-1a76-984d-c97837df7b7a +2024-07-05T13:35:43.559Z [INFO] server-11989612-ef93-a440-3fd6-80d2ff507dc1: pipelining replication: peer="{Voter server-cb9766e1-4150-1a76-984d-c97837df7b7a cb9766e1-4150-1a76-984d-c97837df7b7a}" +2024-07-05T13:35:43.663Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:43.558714721 +0000 UTC m=+9.554636173 (1 nodes), 61.700373ms from start of poll, 65.894171ms from cluster start. Timeout at 2024-07-05 13:35:43.66380176 +0000 UTC m=+9.659723212, 105.087039ms after stability +2024-07-05T13:35:43.664Z [INFO] server-11989612-ef93-a440-3fd6-80d2ff507dc1: updating configuration: command=AddVoter server-id=server-f347f5f9-680d-e3ad-685e-073f6bce0b45 server-addr=f347f5f9-680d-e3ad-685e-073f6bce0b45 servers="[{Suffrage:Voter ID:server-cb9766e1-4150-1a76-984d-c97837df7b7a Address:cb9766e1-4150-1a76-984d-c97837df7b7a} {Suffrage:Voter ID:server-11989612-ef93-a440-3fd6-80d2ff507dc1 Address:11989612-ef93-a440-3fd6-80d2ff507dc1} {Suffrage:Voter ID:server-f347f5f9-680d-e3ad-685e-073f6bce0b45 Address:f347f5f9-680d-e3ad-685e-073f6bce0b45}]" +2024-07-05T13:35:43.664Z [INFO] server-11989612-ef93-a440-3fd6-80d2ff507dc1: added peer, starting replication: peer=server-f347f5f9-680d-e3ad-685e-073f6bce0b45 +2024-07-05T13:35:43.665Z [WARN] server-f347f5f9-680d-e3ad-685e-073f6bce0b45: failed to get previous log: previous-index=3 last-index=0 error="log not found" +2024-07-05T13:35:43.665Z [WARN] server-11989612-ef93-a440-3fd6-80d2ff507dc1: appendEntries rejected, sending older logs: peer="{Voter server-f347f5f9-680d-e3ad-685e-073f6bce0b45 f347f5f9-680d-e3ad-685e-073f6bce0b45}" next=1 +2024-07-05T13:35:43.666Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:43.666Z [INFO] server-11989612-ef93-a440-3fd6-80d2ff507dc1: pipelining replication: peer="{Voter server-f347f5f9-680d-e3ad-685e-073f6bce0b45 f347f5f9-680d-e3ad-685e-073f6bce0b45}" +2024-07-05T13:35:43.772Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:43.666219924 +0000 UTC m=+9.662141376 (1 nodes), 6.417µs from start of poll, 173.399374ms from cluster start. Timeout at 2024-07-05 13:35:43.772164156 +0000 UTC m=+9.768085607, 105.944231ms after stability +2024-07-05T13:35:43.772Z [INFO] server-11989612-ef93-a440-3fd6-80d2ff507dc1: aborting pipeline replication: peer="{Voter server-f347f5f9-680d-e3ad-685e-073f6bce0b45 f347f5f9-680d-e3ad-685e-073f6bce0b45}" +2024-07-05T13:35:43.772Z [INFO] server-11989612-ef93-a440-3fd6-80d2ff507dc1: aborting pipeline replication: peer="{Voter server-cb9766e1-4150-1a76-984d-c97837df7b7a cb9766e1-4150-1a76-984d-c97837df7b7a}" +--- PASS: TestRaft_JoinNode (0.29s) === RUN TestRaft_JoinNode_ConfigStore -2024-07-05T13:29:28.486Z [INFO] server-9213db92-ce58-540c-9d4b-8f8fa476445c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9213db92-ce58-540c-9d4b-8f8fa476445c Address:9213db92-ce58-540c-9d4b-8f8fa476445c}]" -2024-07-05T13:29:28.486Z [INFO] server-0c2fc601-b118-44b4-396d-6ec35e30a98f: initial configuration: index=0 servers=[] -2024-07-05T13:29:28.486Z [INFO] server-b060303f-7729-9817-f6ef-8da44aeaf73a: initial configuration: index=0 servers=[] -2024-07-05T13:29:28.486Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:28.486Z [INFO] server-9213db92-ce58-540c-9d4b-8f8fa476445c: entering follower state: follower="Node at 9213db92-ce58-540c-9d4b-8f8fa476445c [Follower]" leader-address= leader-id= -2024-07-05T13:29:28.487Z [INFO] server-0c2fc601-b118-44b4-396d-6ec35e30a98f: entering follower state: follower="Node at 0c2fc601-b118-44b4-396d-6ec35e30a98f [Follower]" leader-address= leader-id= -2024-07-05T13:29:28.487Z [INFO] server-b060303f-7729-9817-f6ef-8da44aeaf73a: entering follower state: follower="Node at b060303f-7729-9817-f6ef-8da44aeaf73a [Follower]" leader-address= leader-id= -2024-07-05T13:29:28.572Z [WARN] server-b060303f-7729-9817-f6ef-8da44aeaf73a: no known peers, aborting election -2024-07-05T13:29:28.588Z [WARN] server-0c2fc601-b118-44b4-396d-6ec35e30a98f: no known peers, aborting election -2024-07-05T13:29:28.588Z [WARN] server-9213db92-ce58-540c-9d4b-8f8fa476445c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:28.588Z [INFO] server-9213db92-ce58-540c-9d4b-8f8fa476445c: entering candidate state: node="Node at 9213db92-ce58-540c-9d4b-8f8fa476445c [Candidate]" term=2 -2024-07-05T13:29:28.588Z [INFO] server-9213db92-ce58-540c-9d4b-8f8fa476445c: election won: term=2 tally=1 -2024-07-05T13:29:28.588Z [INFO] server-9213db92-ce58-540c-9d4b-8f8fa476445c: entering leader state: leader="Node at 9213db92-ce58-540c-9d4b-8f8fa476445c [Leader]" -2024-07-05T13:29:28.694Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:28.588652184 +0000 UTC m=+9.358562859 (1 nodes), 101.711952ms from start of poll, 102.769355ms from cluster start. Timeout at 2024-07-05 13:29:28.694053566 +0000 UTC m=+9.463964241, 105.401382ms after stability -2024-07-05T13:29:28.694Z [INFO] server-9213db92-ce58-540c-9d4b-8f8fa476445c: updating configuration: command=AddVoter server-id=server-0c2fc601-b118-44b4-396d-6ec35e30a98f server-addr=0c2fc601-b118-44b4-396d-6ec35e30a98f servers="[{Suffrage:Voter ID:server-9213db92-ce58-540c-9d4b-8f8fa476445c Address:9213db92-ce58-540c-9d4b-8f8fa476445c} {Suffrage:Voter ID:server-0c2fc601-b118-44b4-396d-6ec35e30a98f Address:0c2fc601-b118-44b4-396d-6ec35e30a98f}]" -2024-07-05T13:29:28.694Z [INFO] server-9213db92-ce58-540c-9d4b-8f8fa476445c: added peer, starting replication: peer=server-0c2fc601-b118-44b4-396d-6ec35e30a98f -2024-07-05T13:29:28.694Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:28.694Z [WARN] server-0c2fc601-b118-44b4-396d-6ec35e30a98f: failed to get previous log: previous-index=3 last-index=0 error="log not found" -2024-07-05T13:29:28.694Z [WARN] server-9213db92-ce58-540c-9d4b-8f8fa476445c: appendEntries rejected, sending older logs: peer="{Voter server-0c2fc601-b118-44b4-396d-6ec35e30a98f 0c2fc601-b118-44b4-396d-6ec35e30a98f}" next=1 -2024-07-05T13:29:28.694Z [INFO] server-9213db92-ce58-540c-9d4b-8f8fa476445c: pipelining replication: peer="{Voter server-0c2fc601-b118-44b4-396d-6ec35e30a98f 0c2fc601-b118-44b4-396d-6ec35e30a98f}" -2024-07-05T13:29:28.800Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:28.694608227 +0000 UTC m=+9.464518902 (1 nodes), 1.48µs from start of poll, 208.725398ms from cluster start. Timeout at 2024-07-05 13:29:28.800244561 +0000 UTC m=+9.570155236, 105.636334ms after stability -2024-07-05T13:29:28.800Z [INFO] server-9213db92-ce58-540c-9d4b-8f8fa476445c: updating configuration: command=AddVoter server-id=server-b060303f-7729-9817-f6ef-8da44aeaf73a server-addr=b060303f-7729-9817-f6ef-8da44aeaf73a servers="[{Suffrage:Voter ID:server-9213db92-ce58-540c-9d4b-8f8fa476445c Address:9213db92-ce58-540c-9d4b-8f8fa476445c} {Suffrage:Voter ID:server-0c2fc601-b118-44b4-396d-6ec35e30a98f Address:0c2fc601-b118-44b4-396d-6ec35e30a98f} {Suffrage:Voter ID:server-b060303f-7729-9817-f6ef-8da44aeaf73a Address:b060303f-7729-9817-f6ef-8da44aeaf73a}]" -2024-07-05T13:29:28.800Z [INFO] server-9213db92-ce58-540c-9d4b-8f8fa476445c: added peer, starting replication: peer=server-b060303f-7729-9817-f6ef-8da44aeaf73a -2024-07-05T13:29:28.800Z [WARN] server-b060303f-7729-9817-f6ef-8da44aeaf73a: failed to get previous log: previous-index=4 last-index=0 error="log not found" -2024-07-05T13:29:28.800Z [WARN] server-9213db92-ce58-540c-9d4b-8f8fa476445c: appendEntries rejected, sending older logs: peer="{Voter server-b060303f-7729-9817-f6ef-8da44aeaf73a b060303f-7729-9817-f6ef-8da44aeaf73a}" next=1 -2024-07-05T13:29:28.800Z [INFO] server-9213db92-ce58-540c-9d4b-8f8fa476445c: pipelining replication: peer="{Voter server-b060303f-7729-9817-f6ef-8da44aeaf73a b060303f-7729-9817-f6ef-8da44aeaf73a}" -2024-07-05T13:29:28.801Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:28.907Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:28.801260165 +0000 UTC m=+9.571170840 (1 nodes), 1.6µs from start of poll, 315.377336ms from cluster start. Timeout at 2024-07-05 13:29:28.907125171 +0000 UTC m=+9.677035846, 105.865006ms after stability -2024-07-05T13:29:28.907Z [INFO] server-9213db92-ce58-540c-9d4b-8f8fa476445c: aborting pipeline replication: peer="{Voter server-b060303f-7729-9817-f6ef-8da44aeaf73a b060303f-7729-9817-f6ef-8da44aeaf73a}" -2024-07-05T13:29:28.907Z [INFO] server-9213db92-ce58-540c-9d4b-8f8fa476445c: aborting pipeline replication: peer="{Voter server-0c2fc601-b118-44b4-396d-6ec35e30a98f 0c2fc601-b118-44b4-396d-6ec35e30a98f}" ---- PASS: TestRaft_JoinNode_ConfigStore (0.42s) +2024-07-05T13:35:43.780Z [INFO] server-d337d5b8-2bb5-16fb-e206-13d85176a130: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d337d5b8-2bb5-16fb-e206-13d85176a130 Address:d337d5b8-2bb5-16fb-e206-13d85176a130}]" +2024-07-05T13:35:43.780Z [INFO] server-d337d5b8-2bb5-16fb-e206-13d85176a130: entering follower state: follower="Node at d337d5b8-2bb5-16fb-e206-13d85176a130 [Follower]" leader-address= leader-id= +2024-07-05T13:35:43.782Z [INFO] server-a3d3965e-424d-9a1e-8a48-92d822834a66: initial configuration: index=0 servers=[] +2024-07-05T13:35:43.782Z [INFO] server-a3d3965e-424d-9a1e-8a48-92d822834a66: entering follower state: follower="Node at a3d3965e-424d-9a1e-8a48-92d822834a66 [Follower]" leader-address= leader-id= +2024-07-05T13:35:43.785Z [INFO] server-c081afb1-f5ed-4e02-d2a2-1d44de38b517: initial configuration: index=0 servers=[] +2024-07-05T13:35:43.785Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:43.785Z [INFO] server-c081afb1-f5ed-4e02-d2a2-1d44de38b517: entering follower state: follower="Node at c081afb1-f5ed-4e02-d2a2-1d44de38b517 [Follower]" leader-address= leader-id= +2024-07-05T13:35:43.857Z [WARN] server-a3d3965e-424d-9a1e-8a48-92d822834a66: no known peers, aborting election +2024-07-05T13:35:43.859Z [WARN] server-c081afb1-f5ed-4e02-d2a2-1d44de38b517: no known peers, aborting election +2024-07-05T13:35:43.860Z [WARN] server-d337d5b8-2bb5-16fb-e206-13d85176a130: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:43.860Z [INFO] server-d337d5b8-2bb5-16fb-e206-13d85176a130: entering candidate state: node="Node at d337d5b8-2bb5-16fb-e206-13d85176a130 [Candidate]" term=2 +2024-07-05T13:35:43.861Z [INFO] server-d337d5b8-2bb5-16fb-e206-13d85176a130: election won: term=2 tally=1 +2024-07-05T13:35:43.861Z [INFO] server-d337d5b8-2bb5-16fb-e206-13d85176a130: entering leader state: leader="Node at d337d5b8-2bb5-16fb-e206-13d85176a130 [Leader]" +2024-07-05T13:35:43.967Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:43.861708561 +0000 UTC m=+9.857630305 (1 nodes), 76.250191ms from start of poll, 82.071168ms from cluster start. Timeout at 2024-07-05 13:35:43.967081428 +0000 UTC m=+9.963002880, 105.372575ms after stability +2024-07-05T13:35:43.967Z [INFO] server-d337d5b8-2bb5-16fb-e206-13d85176a130: updating configuration: command=AddVoter server-id=server-a3d3965e-424d-9a1e-8a48-92d822834a66 server-addr=a3d3965e-424d-9a1e-8a48-92d822834a66 servers="[{Suffrage:Voter ID:server-d337d5b8-2bb5-16fb-e206-13d85176a130 Address:d337d5b8-2bb5-16fb-e206-13d85176a130} {Suffrage:Voter ID:server-a3d3965e-424d-9a1e-8a48-92d822834a66 Address:a3d3965e-424d-9a1e-8a48-92d822834a66}]" +2024-07-05T13:35:43.968Z [INFO] server-d337d5b8-2bb5-16fb-e206-13d85176a130: added peer, starting replication: peer=server-a3d3965e-424d-9a1e-8a48-92d822834a66 +2024-07-05T13:35:43.968Z [WARN] server-a3d3965e-424d-9a1e-8a48-92d822834a66: failed to get previous log: previous-index=3 last-index=0 error="log not found" +2024-07-05T13:35:43.968Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:43.968Z [WARN] server-d337d5b8-2bb5-16fb-e206-13d85176a130: appendEntries rejected, sending older logs: peer="{Voter server-a3d3965e-424d-9a1e-8a48-92d822834a66 a3d3965e-424d-9a1e-8a48-92d822834a66}" next=1 +2024-07-05T13:35:43.969Z [INFO] server-d337d5b8-2bb5-16fb-e206-13d85176a130: pipelining replication: peer="{Voter server-a3d3965e-424d-9a1e-8a48-92d822834a66 a3d3965e-424d-9a1e-8a48-92d822834a66}" +2024-07-05T13:35:44.074Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:43.968731648 +0000 UTC m=+9.964653100 (1 nodes), 5.833µs from start of poll, 189.093963ms from cluster start. Timeout at 2024-07-05 13:35:44.07446355 +0000 UTC m=+10.070385002, 105.731902ms after stability +2024-07-05T13:35:44.074Z [INFO] server-d337d5b8-2bb5-16fb-e206-13d85176a130: updating configuration: command=AddVoter server-id=server-c081afb1-f5ed-4e02-d2a2-1d44de38b517 server-addr=c081afb1-f5ed-4e02-d2a2-1d44de38b517 servers="[{Suffrage:Voter ID:server-d337d5b8-2bb5-16fb-e206-13d85176a130 Address:d337d5b8-2bb5-16fb-e206-13d85176a130} {Suffrage:Voter ID:server-a3d3965e-424d-9a1e-8a48-92d822834a66 Address:a3d3965e-424d-9a1e-8a48-92d822834a66} {Suffrage:Voter ID:server-c081afb1-f5ed-4e02-d2a2-1d44de38b517 Address:c081afb1-f5ed-4e02-d2a2-1d44de38b517}]" +2024-07-05T13:35:44.075Z [INFO] server-d337d5b8-2bb5-16fb-e206-13d85176a130: added peer, starting replication: peer=server-c081afb1-f5ed-4e02-d2a2-1d44de38b517 +2024-07-05T13:35:44.075Z [WARN] server-c081afb1-f5ed-4e02-d2a2-1d44de38b517: failed to get previous log: previous-index=4 last-index=0 error="log not found" +2024-07-05T13:35:44.076Z [WARN] server-d337d5b8-2bb5-16fb-e206-13d85176a130: appendEntries rejected, sending older logs: peer="{Voter server-c081afb1-f5ed-4e02-d2a2-1d44de38b517 c081afb1-f5ed-4e02-d2a2-1d44de38b517}" next=1 +2024-07-05T13:35:44.076Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:44.077Z [INFO] server-d337d5b8-2bb5-16fb-e206-13d85176a130: pipelining replication: peer="{Voter server-c081afb1-f5ed-4e02-d2a2-1d44de38b517 c081afb1-f5ed-4e02-d2a2-1d44de38b517}" +2024-07-05T13:35:44.182Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:44.076785466 +0000 UTC m=+10.072706918 (1 nodes), 5.834µs from start of poll, 297.147781ms from cluster start. Timeout at 2024-07-05 13:35:44.18203671 +0000 UTC m=+10.177958162, 105.251244ms after stability +2024-07-05T13:35:44.182Z [INFO] server-d337d5b8-2bb5-16fb-e206-13d85176a130: aborting pipeline replication: peer="{Voter server-c081afb1-f5ed-4e02-d2a2-1d44de38b517 c081afb1-f5ed-4e02-d2a2-1d44de38b517}" +2024-07-05T13:35:44.182Z [INFO] server-d337d5b8-2bb5-16fb-e206-13d85176a130: aborting pipeline replication: peer="{Voter server-a3d3965e-424d-9a1e-8a48-92d822834a66 a3d3965e-424d-9a1e-8a48-92d822834a66}" +--- PASS: TestRaft_JoinNode_ConfigStore (0.41s) === RUN TestRaft_RemoveFollower -2024-07-05T13:29:28.910Z [INFO] server-7d24dd2f-77f0-0a32-c8d3-7366dac882dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d24dd2f-77f0-0a32-c8d3-7366dac882dc Address:7d24dd2f-77f0-0a32-c8d3-7366dac882dc} {Suffrage:Voter ID:server-410e220c-a4c7-f983-5dc3-01151652ea0b Address:410e220c-a4c7-f983-5dc3-01151652ea0b} {Suffrage:Voter ID:server-c8344457-9ea7-0d5c-137a-0c250eb51904 Address:c8344457-9ea7-0d5c-137a-0c250eb51904}]" -2024-07-05T13:29:28.910Z [INFO] server-410e220c-a4c7-f983-5dc3-01151652ea0b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d24dd2f-77f0-0a32-c8d3-7366dac882dc Address:7d24dd2f-77f0-0a32-c8d3-7366dac882dc} {Suffrage:Voter ID:server-410e220c-a4c7-f983-5dc3-01151652ea0b Address:410e220c-a4c7-f983-5dc3-01151652ea0b} {Suffrage:Voter ID:server-c8344457-9ea7-0d5c-137a-0c250eb51904 Address:c8344457-9ea7-0d5c-137a-0c250eb51904}]" -2024-07-05T13:29:28.911Z [INFO] server-c8344457-9ea7-0d5c-137a-0c250eb51904: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d24dd2f-77f0-0a32-c8d3-7366dac882dc Address:7d24dd2f-77f0-0a32-c8d3-7366dac882dc} {Suffrage:Voter ID:server-410e220c-a4c7-f983-5dc3-01151652ea0b Address:410e220c-a4c7-f983-5dc3-01151652ea0b} {Suffrage:Voter ID:server-c8344457-9ea7-0d5c-137a-0c250eb51904 Address:c8344457-9ea7-0d5c-137a-0c250eb51904}]" -2024-07-05T13:29:28.911Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:28.911Z [INFO] server-7d24dd2f-77f0-0a32-c8d3-7366dac882dc: entering follower state: follower="Node at 7d24dd2f-77f0-0a32-c8d3-7366dac882dc [Follower]" leader-address= leader-id= -2024-07-05T13:29:28.911Z [INFO] server-410e220c-a4c7-f983-5dc3-01151652ea0b: entering follower state: follower="Node at 410e220c-a4c7-f983-5dc3-01151652ea0b [Follower]" leader-address= leader-id= -2024-07-05T13:29:28.911Z [INFO] server-c8344457-9ea7-0d5c-137a-0c250eb51904: entering follower state: follower="Node at c8344457-9ea7-0d5c-137a-0c250eb51904 [Follower]" leader-address= leader-id= -2024-07-05T13:29:28.962Z [WARN] server-410e220c-a4c7-f983-5dc3-01151652ea0b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:28.962Z [INFO] server-410e220c-a4c7-f983-5dc3-01151652ea0b: entering candidate state: node="Node at 410e220c-a4c7-f983-5dc3-01151652ea0b [Candidate]" term=2 -2024-07-05T13:29:28.962Z [INFO] server-410e220c-a4c7-f983-5dc3-01151652ea0b: election won: term=2 tally=2 -2024-07-05T13:29:28.962Z [INFO] server-410e220c-a4c7-f983-5dc3-01151652ea0b: entering leader state: leader="Node at 410e220c-a4c7-f983-5dc3-01151652ea0b [Leader]" -2024-07-05T13:29:28.962Z [INFO] server-410e220c-a4c7-f983-5dc3-01151652ea0b: added peer, starting replication: peer=server-7d24dd2f-77f0-0a32-c8d3-7366dac882dc -2024-07-05T13:29:28.962Z [INFO] server-410e220c-a4c7-f983-5dc3-01151652ea0b: added peer, starting replication: peer=server-c8344457-9ea7-0d5c-137a-0c250eb51904 -2024-07-05T13:29:28.963Z [INFO] server-410e220c-a4c7-f983-5dc3-01151652ea0b: pipelining replication: peer="{Voter server-c8344457-9ea7-0d5c-137a-0c250eb51904 c8344457-9ea7-0d5c-137a-0c250eb51904}" -2024-07-05T13:29:28.963Z [INFO] server-410e220c-a4c7-f983-5dc3-01151652ea0b: pipelining replication: peer="{Voter server-7d24dd2f-77f0-0a32-c8d3-7366dac882dc 7d24dd2f-77f0-0a32-c8d3-7366dac882dc}" -2024-07-05T13:29:29.071Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:28.963357679 +0000 UTC m=+9.733268354 (1 nodes), 52.211449ms from start of poll, 52.943423ms from cluster start. Timeout at 2024-07-05 13:29:29.071127978 +0000 UTC m=+9.841038653, 107.770299ms after stability -2024-07-05T13:29:29.076Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:29:29.182Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:29:29.076742461 +0000 UTC m=+9.846653136 (2 nodes), 2.56µs from start of poll, 166.328205ms from cluster start. Timeout at 2024-07-05 13:29:29.182886338 +0000 UTC m=+9.952797013, 106.143877ms after stability -2024-07-05T13:29:29.183Z [INFO] server-410e220c-a4c7-f983-5dc3-01151652ea0b: updating configuration: command=RemoveServer server-id=server-7d24dd2f-77f0-0a32-c8d3-7366dac882dc server-addr= servers="[{Suffrage:Voter ID:server-410e220c-a4c7-f983-5dc3-01151652ea0b Address:410e220c-a4c7-f983-5dc3-01151652ea0b} {Suffrage:Voter ID:server-c8344457-9ea7-0d5c-137a-0c250eb51904 Address:c8344457-9ea7-0d5c-137a-0c250eb51904}]" -2024-07-05T13:29:29.183Z [INFO] server-410e220c-a4c7-f983-5dc3-01151652ea0b: removed peer, stopping replication: peer=server-7d24dd2f-77f0-0a32-c8d3-7366dac882dc last-index=3 -2024-07-05T13:29:29.183Z [INFO] server-410e220c-a4c7-f983-5dc3-01151652ea0b: aborting pipeline replication: peer="{Voter server-7d24dd2f-77f0-0a32-c8d3-7366dac882dc 7d24dd2f-77f0-0a32-c8d3-7366dac882dc}" -2024-07-05T13:29:29.288Z [INFO] server-410e220c-a4c7-f983-5dc3-01151652ea0b: aborting pipeline replication: peer="{Voter server-c8344457-9ea7-0d5c-137a-0c250eb51904 c8344457-9ea7-0d5c-137a-0c250eb51904}" ---- PASS: TestRaft_RemoveFollower (0.38s) +2024-07-05T13:35:44.193Z [INFO] server-aefcd4d3-a195-acc7-25f0-d5249f8db387: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aefcd4d3-a195-acc7-25f0-d5249f8db387 Address:aefcd4d3-a195-acc7-25f0-d5249f8db387} {Suffrage:Voter ID:server-aee8ece9-bbc9-8d64-040b-458474d1ebe2 Address:aee8ece9-bbc9-8d64-040b-458474d1ebe2} {Suffrage:Voter ID:server-3ed49cc5-72f2-e3a9-4c66-abdfc04c4ec8 Address:3ed49cc5-72f2-e3a9-4c66-abdfc04c4ec8}]" +2024-07-05T13:35:44.194Z [INFO] server-aefcd4d3-a195-acc7-25f0-d5249f8db387: entering follower state: follower="Node at aefcd4d3-a195-acc7-25f0-d5249f8db387 [Follower]" leader-address= leader-id= +2024-07-05T13:35:44.194Z [INFO] server-aee8ece9-bbc9-8d64-040b-458474d1ebe2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aefcd4d3-a195-acc7-25f0-d5249f8db387 Address:aefcd4d3-a195-acc7-25f0-d5249f8db387} {Suffrage:Voter ID:server-aee8ece9-bbc9-8d64-040b-458474d1ebe2 Address:aee8ece9-bbc9-8d64-040b-458474d1ebe2} {Suffrage:Voter ID:server-3ed49cc5-72f2-e3a9-4c66-abdfc04c4ec8 Address:3ed49cc5-72f2-e3a9-4c66-abdfc04c4ec8}]" +2024-07-05T13:35:44.195Z [INFO] server-aee8ece9-bbc9-8d64-040b-458474d1ebe2: entering follower state: follower="Node at aee8ece9-bbc9-8d64-040b-458474d1ebe2 [Follower]" leader-address= leader-id= +2024-07-05T13:35:44.195Z [INFO] server-3ed49cc5-72f2-e3a9-4c66-abdfc04c4ec8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aefcd4d3-a195-acc7-25f0-d5249f8db387 Address:aefcd4d3-a195-acc7-25f0-d5249f8db387} {Suffrage:Voter ID:server-aee8ece9-bbc9-8d64-040b-458474d1ebe2 Address:aee8ece9-bbc9-8d64-040b-458474d1ebe2} {Suffrage:Voter ID:server-3ed49cc5-72f2-e3a9-4c66-abdfc04c4ec8 Address:3ed49cc5-72f2-e3a9-4c66-abdfc04c4ec8}]" +2024-07-05T13:35:44.195Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:44.196Z [INFO] server-3ed49cc5-72f2-e3a9-4c66-abdfc04c4ec8: entering follower state: follower="Node at 3ed49cc5-72f2-e3a9-4c66-abdfc04c4ec8 [Follower]" leader-address= leader-id= +2024-07-05T13:35:44.264Z [WARN] server-aefcd4d3-a195-acc7-25f0-d5249f8db387: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:44.264Z [INFO] server-aefcd4d3-a195-acc7-25f0-d5249f8db387: entering candidate state: node="Node at aefcd4d3-a195-acc7-25f0-d5249f8db387 [Candidate]" term=2 +2024-07-05T13:35:44.265Z [INFO] server-aefcd4d3-a195-acc7-25f0-d5249f8db387: election won: term=2 tally=2 +2024-07-05T13:35:44.265Z [INFO] server-aefcd4d3-a195-acc7-25f0-d5249f8db387: entering leader state: leader="Node at aefcd4d3-a195-acc7-25f0-d5249f8db387 [Leader]" +2024-07-05T13:35:44.266Z [INFO] server-aefcd4d3-a195-acc7-25f0-d5249f8db387: added peer, starting replication: peer=server-aee8ece9-bbc9-8d64-040b-458474d1ebe2 +2024-07-05T13:35:44.266Z [INFO] server-aefcd4d3-a195-acc7-25f0-d5249f8db387: added peer, starting replication: peer=server-3ed49cc5-72f2-e3a9-4c66-abdfc04c4ec8 +2024-07-05T13:35:44.267Z [INFO] server-aefcd4d3-a195-acc7-25f0-d5249f8db387: pipelining replication: peer="{Voter server-aee8ece9-bbc9-8d64-040b-458474d1ebe2 aee8ece9-bbc9-8d64-040b-458474d1ebe2}" +2024-07-05T13:35:44.267Z [INFO] server-aefcd4d3-a195-acc7-25f0-d5249f8db387: pipelining replication: peer="{Voter server-3ed49cc5-72f2-e3a9-4c66-abdfc04c4ec8 3ed49cc5-72f2-e3a9-4c66-abdfc04c4ec8}" +2024-07-05T13:35:44.371Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:44.266239047 +0000 UTC m=+10.262160790 (1 nodes), 70.258591ms from start of poll, 73.069915ms from cluster start. Timeout at 2024-07-05 13:35:44.371541623 +0000 UTC m=+10.367463075, 105.302285ms after stability +2024-07-05T13:35:44.377Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:35:44.483Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:35:44.378026422 +0000 UTC m=+10.373947873 (2 nodes), 7.875µs from start of poll, 184.856998ms from cluster start. Timeout at 2024-07-05 13:35:44.483050462 +0000 UTC m=+10.478971913, 105.02404ms after stability +2024-07-05T13:35:44.483Z [INFO] server-aefcd4d3-a195-acc7-25f0-d5249f8db387: updating configuration: command=RemoveServer server-id=server-aee8ece9-bbc9-8d64-040b-458474d1ebe2 server-addr= servers="[{Suffrage:Voter ID:server-aefcd4d3-a195-acc7-25f0-d5249f8db387 Address:aefcd4d3-a195-acc7-25f0-d5249f8db387} {Suffrage:Voter ID:server-3ed49cc5-72f2-e3a9-4c66-abdfc04c4ec8 Address:3ed49cc5-72f2-e3a9-4c66-abdfc04c4ec8}]" +2024-07-05T13:35:44.483Z [INFO] server-aefcd4d3-a195-acc7-25f0-d5249f8db387: removed peer, stopping replication: peer=server-aee8ece9-bbc9-8d64-040b-458474d1ebe2 last-index=3 +2024-07-05T13:35:44.484Z [INFO] server-aefcd4d3-a195-acc7-25f0-d5249f8db387: aborting pipeline replication: peer="{Voter server-aee8ece9-bbc9-8d64-040b-458474d1ebe2 aee8ece9-bbc9-8d64-040b-458474d1ebe2}" +2024-07-05T13:35:44.588Z [WARN] server-aee8ece9-bbc9-8d64-040b-458474d1ebe2: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election +2024-07-05T13:35:44.590Z [INFO] server-aefcd4d3-a195-acc7-25f0-d5249f8db387: aborting pipeline replication: peer="{Voter server-3ed49cc5-72f2-e3a9-4c66-abdfc04c4ec8 3ed49cc5-72f2-e3a9-4c66-abdfc04c4ec8}" +--- PASS: TestRaft_RemoveFollower (0.41s) === RUN TestRaft_RemoveLeader -2024-07-05T13:29:29.290Z [INFO] server-230c1c63-758f-924e-a8a1-3aaa62f2c653: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-230c1c63-758f-924e-a8a1-3aaa62f2c653 Address:230c1c63-758f-924e-a8a1-3aaa62f2c653} {Suffrage:Voter ID:server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c Address:6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c} {Suffrage:Voter ID:server-3296db68-1621-2e3d-d957-4a3869d788cb Address:3296db68-1621-2e3d-d957-4a3869d788cb}]" -2024-07-05T13:29:29.291Z [INFO] server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-230c1c63-758f-924e-a8a1-3aaa62f2c653 Address:230c1c63-758f-924e-a8a1-3aaa62f2c653} {Suffrage:Voter ID:server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c Address:6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c} {Suffrage:Voter ID:server-3296db68-1621-2e3d-d957-4a3869d788cb Address:3296db68-1621-2e3d-d957-4a3869d788cb}]" -2024-07-05T13:29:29.291Z [INFO] server-3296db68-1621-2e3d-d957-4a3869d788cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-230c1c63-758f-924e-a8a1-3aaa62f2c653 Address:230c1c63-758f-924e-a8a1-3aaa62f2c653} {Suffrage:Voter ID:server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c Address:6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c} {Suffrage:Voter ID:server-3296db68-1621-2e3d-d957-4a3869d788cb Address:3296db68-1621-2e3d-d957-4a3869d788cb}]" -2024-07-05T13:29:29.291Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:29.291Z [INFO] server-230c1c63-758f-924e-a8a1-3aaa62f2c653: entering follower state: follower="Node at 230c1c63-758f-924e-a8a1-3aaa62f2c653 [Follower]" leader-address= leader-id= -2024-07-05T13:29:29.291Z [INFO] server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c: entering follower state: follower="Node at 6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c [Follower]" leader-address= leader-id= -2024-07-05T13:29:29.291Z [INFO] server-3296db68-1621-2e3d-d957-4a3869d788cb: entering follower state: follower="Node at 3296db68-1621-2e3d-d957-4a3869d788cb [Follower]" leader-address= leader-id= -2024-07-05T13:29:29.350Z [WARN] server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:29.350Z [INFO] server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c: entering candidate state: node="Node at 6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c [Candidate]" term=2 -2024-07-05T13:29:29.350Z [INFO] server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c: election won: term=2 tally=2 -2024-07-05T13:29:29.350Z [INFO] server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c: entering leader state: leader="Node at 6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c [Leader]" -2024-07-05T13:29:29.350Z [INFO] server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c: added peer, starting replication: peer=server-230c1c63-758f-924e-a8a1-3aaa62f2c653 -2024-07-05T13:29:29.350Z [INFO] server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c: added peer, starting replication: peer=server-3296db68-1621-2e3d-d957-4a3869d788cb -2024-07-05T13:29:29.350Z [INFO] server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c: pipelining replication: peer="{Voter server-3296db68-1621-2e3d-d957-4a3869d788cb 3296db68-1621-2e3d-d957-4a3869d788cb}" -2024-07-05T13:29:29.350Z [INFO] server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c: pipelining replication: peer="{Voter server-230c1c63-758f-924e-a8a1-3aaa62f2c653 230c1c63-758f-924e-a8a1-3aaa62f2c653}" -2024-07-05T13:29:29.456Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:29.350940362 +0000 UTC m=+10.120851037 (1 nodes), 59.533431ms from start of poll, 60.158129ms from cluster start. Timeout at 2024-07-05 13:29:29.456781049 +0000 UTC m=+10.226691724, 105.840687ms after stability -2024-07-05T13:29:29.462Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:29:29.568Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:29:29.462202779 +0000 UTC m=+10.232113454 (2 nodes), 2.2µs from start of poll, 171.420546ms from cluster start. Timeout at 2024-07-05 13:29:29.568859117 +0000 UTC m=+10.338769832, 106.656378ms after stability -2024-07-05T13:29:29.569Z [INFO] server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c: updating configuration: command=RemoveServer server-id=server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c server-addr= servers="[{Suffrage:Voter ID:server-230c1c63-758f-924e-a8a1-3aaa62f2c653 Address:230c1c63-758f-924e-a8a1-3aaa62f2c653} {Suffrage:Voter ID:server-3296db68-1621-2e3d-d957-4a3869d788cb Address:3296db68-1621-2e3d-d957-4a3869d788cb}]" -2024-07-05T13:29:29.569Z [INFO] server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c: removed ourself, shutting down -2024-07-05T13:29:29.569Z [INFO] server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c: aborting pipeline replication: peer="{Voter server-3296db68-1621-2e3d-d957-4a3869d788cb 3296db68-1621-2e3d-d957-4a3869d788cb}" -2024-07-05T13:29:29.569Z [INFO] server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c: aborting pipeline replication: peer="{Voter server-230c1c63-758f-924e-a8a1-3aaa62f2c653 230c1c63-758f-924e-a8a1-3aaa62f2c653}" -2024-07-05T13:29:29.648Z [WARN] server-3296db68-1621-2e3d-d957-4a3869d788cb: heartbeat timeout reached, starting election: last-leader-addr=6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c last-leader-id=server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c -2024-07-05T13:29:29.648Z [INFO] server-3296db68-1621-2e3d-d957-4a3869d788cb: entering candidate state: node="Node at 3296db68-1621-2e3d-d957-4a3869d788cb [Candidate]" term=3 -2024-07-05T13:29:29.648Z [WARN] server-230c1c63-758f-924e-a8a1-3aaa62f2c653: rejecting vote request since we have a leader: from=3296db68-1621-2e3d-d957-4a3869d788cb leader=6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c leader-id=server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c -2024-07-05T13:29:29.665Z [WARN] server-230c1c63-758f-924e-a8a1-3aaa62f2c653: heartbeat timeout reached, starting election: last-leader-addr=6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c last-leader-id=server-6cc0eb05-ab41-9fdc-c09d-c3b2fb59490c -2024-07-05T13:29:29.665Z [INFO] server-230c1c63-758f-924e-a8a1-3aaa62f2c653: entering candidate state: node="Node at 230c1c63-758f-924e-a8a1-3aaa62f2c653 [Candidate]" term=3 -2024-07-05T13:29:29.665Z [INFO] server-3296db68-1621-2e3d-d957-4a3869d788cb: duplicate requestVote for same term: term=3 -2024-07-05T13:29:29.730Z [WARN] server-3296db68-1621-2e3d-d957-4a3869d788cb: Election timeout reached, restarting election -2024-07-05T13:29:29.730Z [INFO] server-3296db68-1621-2e3d-d957-4a3869d788cb: entering candidate state: node="Node at 3296db68-1621-2e3d-d957-4a3869d788cb [Candidate]" term=4 -2024-07-05T13:29:29.730Z [INFO] server-230c1c63-758f-924e-a8a1-3aaa62f2c653: entering follower state: follower="Node at 230c1c63-758f-924e-a8a1-3aaa62f2c653 [Follower]" leader-address= leader-id= -2024-07-05T13:29:29.731Z [INFO] server-3296db68-1621-2e3d-d957-4a3869d788cb: election won: term=4 tally=2 -2024-07-05T13:29:29.731Z [INFO] server-3296db68-1621-2e3d-d957-4a3869d788cb: entering leader state: leader="Node at 3296db68-1621-2e3d-d957-4a3869d788cb [Leader]" -2024-07-05T13:29:29.731Z [INFO] server-3296db68-1621-2e3d-d957-4a3869d788cb: added peer, starting replication: peer=server-230c1c63-758f-924e-a8a1-3aaa62f2c653 -2024-07-05T13:29:29.731Z [INFO] server-3296db68-1621-2e3d-d957-4a3869d788cb: pipelining replication: peer="{Voter server-230c1c63-758f-924e-a8a1-3aaa62f2c653 230c1c63-758f-924e-a8a1-3aaa62f2c653}" -2024-07-05T13:29:29.782Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:29.887Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:29.782332229 +0000 UTC m=+10.552242904 (1 nodes), 90.837µs from start of poll, 491.549996ms from cluster start. Timeout at 2024-07-05 13:29:29.887622695 +0000 UTC m=+10.657533370, 105.290466ms after stability -2024-07-05T13:29:29.887Z [INFO] server-3296db68-1621-2e3d-d957-4a3869d788cb: aborting pipeline replication: peer="{Voter server-230c1c63-758f-924e-a8a1-3aaa62f2c653 230c1c63-758f-924e-a8a1-3aaa62f2c653}" ---- PASS: TestRaft_RemoveLeader (0.60s) +2024-07-05T13:35:44.608Z [INFO] server-a2dec55a-4a20-7ae4-41c7-054e74bb7526: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a2dec55a-4a20-7ae4-41c7-054e74bb7526 Address:a2dec55a-4a20-7ae4-41c7-054e74bb7526} {Suffrage:Voter ID:server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34 Address:5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34} {Suffrage:Voter ID:server-d651c130-76c7-27d2-aa7e-cb454000b9bc Address:d651c130-76c7-27d2-aa7e-cb454000b9bc}]" +2024-07-05T13:35:44.608Z [INFO] server-a2dec55a-4a20-7ae4-41c7-054e74bb7526: entering follower state: follower="Node at a2dec55a-4a20-7ae4-41c7-054e74bb7526 [Follower]" leader-address= leader-id= +2024-07-05T13:35:44.609Z [INFO] server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a2dec55a-4a20-7ae4-41c7-054e74bb7526 Address:a2dec55a-4a20-7ae4-41c7-054e74bb7526} {Suffrage:Voter ID:server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34 Address:5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34} {Suffrage:Voter ID:server-d651c130-76c7-27d2-aa7e-cb454000b9bc Address:d651c130-76c7-27d2-aa7e-cb454000b9bc}]" +2024-07-05T13:35:44.609Z [INFO] server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34: entering follower state: follower="Node at 5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34 [Follower]" leader-address= leader-id= +2024-07-05T13:35:44.610Z [INFO] server-d651c130-76c7-27d2-aa7e-cb454000b9bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a2dec55a-4a20-7ae4-41c7-054e74bb7526 Address:a2dec55a-4a20-7ae4-41c7-054e74bb7526} {Suffrage:Voter ID:server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34 Address:5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34} {Suffrage:Voter ID:server-d651c130-76c7-27d2-aa7e-cb454000b9bc Address:d651c130-76c7-27d2-aa7e-cb454000b9bc}]" +2024-07-05T13:35:44.610Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:44.610Z [INFO] server-d651c130-76c7-27d2-aa7e-cb454000b9bc: entering follower state: follower="Node at d651c130-76c7-27d2-aa7e-cb454000b9bc [Follower]" leader-address= leader-id= +2024-07-05T13:35:44.674Z [WARN] server-d651c130-76c7-27d2-aa7e-cb454000b9bc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:44.675Z [INFO] server-d651c130-76c7-27d2-aa7e-cb454000b9bc: entering candidate state: node="Node at d651c130-76c7-27d2-aa7e-cb454000b9bc [Candidate]" term=2 +2024-07-05T13:35:44.675Z [WARN] server-a2dec55a-4a20-7ae4-41c7-054e74bb7526: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:44.675Z [INFO] server-a2dec55a-4a20-7ae4-41c7-054e74bb7526: entering candidate state: node="Node at a2dec55a-4a20-7ae4-41c7-054e74bb7526 [Candidate]" term=2 +2024-07-05T13:35:44.675Z [INFO] server-a2dec55a-4a20-7ae4-41c7-054e74bb7526: duplicate requestVote for same term: term=2 +2024-07-05T13:35:44.675Z [INFO] server-d651c130-76c7-27d2-aa7e-cb454000b9bc: election won: term=2 tally=2 +2024-07-05T13:35:44.675Z [INFO] server-d651c130-76c7-27d2-aa7e-cb454000b9bc: entering leader state: leader="Node at d651c130-76c7-27d2-aa7e-cb454000b9bc [Leader]" +2024-07-05T13:35:44.675Z [INFO] server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34: duplicate requestVote for same term: term=2 +2024-07-05T13:35:44.676Z [INFO] server-d651c130-76c7-27d2-aa7e-cb454000b9bc: added peer, starting replication: peer=server-a2dec55a-4a20-7ae4-41c7-054e74bb7526 +2024-07-05T13:35:44.676Z [INFO] server-d651c130-76c7-27d2-aa7e-cb454000b9bc: added peer, starting replication: peer=server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34 +2024-07-05T13:35:44.676Z [INFO] server-a2dec55a-4a20-7ae4-41c7-054e74bb7526: entering follower state: follower="Node at a2dec55a-4a20-7ae4-41c7-054e74bb7526 [Follower]" leader-address=d651c130-76c7-27d2-aa7e-cb454000b9bc leader-id=server-d651c130-76c7-27d2-aa7e-cb454000b9bc +2024-07-05T13:35:44.676Z [WARN] server-d651c130-76c7-27d2-aa7e-cb454000b9bc: rejecting vote request since we have a leader: from=a2dec55a-4a20-7ae4-41c7-054e74bb7526 leader=d651c130-76c7-27d2-aa7e-cb454000b9bc leader-id=server-d651c130-76c7-27d2-aa7e-cb454000b9bc +2024-07-05T13:35:44.676Z [INFO] server-d651c130-76c7-27d2-aa7e-cb454000b9bc: pipelining replication: peer="{Voter server-a2dec55a-4a20-7ae4-41c7-054e74bb7526 a2dec55a-4a20-7ae4-41c7-054e74bb7526}" +2024-07-05T13:35:44.677Z [INFO] server-d651c130-76c7-27d2-aa7e-cb454000b9bc: pipelining replication: peer="{Voter server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34 5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34}" +2024-07-05T13:35:44.781Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:44.676560177 +0000 UTC m=+10.672481628 (1 nodes), 66.132458ms from start of poll, 69.062488ms from cluster start. Timeout at 2024-07-05 13:35:44.781595008 +0000 UTC m=+10.777516460, 105.034832ms after stability +2024-07-05T13:35:44.787Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:35:44.893Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:35:44.787579607 +0000 UTC m=+10.783500767 (2 nodes), 8.166µs from start of poll, 180.081627ms from cluster start. Timeout at 2024-07-05 13:35:44.89352063 +0000 UTC m=+10.889442374, 105.941607ms after stability +2024-07-05T13:35:44.893Z [INFO] server-d651c130-76c7-27d2-aa7e-cb454000b9bc: updating configuration: command=RemoveServer server-id=server-d651c130-76c7-27d2-aa7e-cb454000b9bc server-addr= servers="[{Suffrage:Voter ID:server-a2dec55a-4a20-7ae4-41c7-054e74bb7526 Address:a2dec55a-4a20-7ae4-41c7-054e74bb7526} {Suffrage:Voter ID:server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34 Address:5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34}]" +2024-07-05T13:35:44.895Z [INFO] server-d651c130-76c7-27d2-aa7e-cb454000b9bc: removed ourself, shutting down +2024-07-05T13:35:44.895Z [INFO] server-d651c130-76c7-27d2-aa7e-cb454000b9bc: aborting pipeline replication: peer="{Voter server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34 5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34}" +2024-07-05T13:35:44.895Z [INFO] server-d651c130-76c7-27d2-aa7e-cb454000b9bc: aborting pipeline replication: peer="{Voter server-a2dec55a-4a20-7ae4-41c7-054e74bb7526 a2dec55a-4a20-7ae4-41c7-054e74bb7526}" +2024-07-05T13:35:44.977Z [WARN] server-a2dec55a-4a20-7ae4-41c7-054e74bb7526: heartbeat timeout reached, starting election: last-leader-addr=d651c130-76c7-27d2-aa7e-cb454000b9bc last-leader-id=server-d651c130-76c7-27d2-aa7e-cb454000b9bc +2024-07-05T13:35:44.977Z [INFO] server-a2dec55a-4a20-7ae4-41c7-054e74bb7526: entering candidate state: node="Node at a2dec55a-4a20-7ae4-41c7-054e74bb7526 [Candidate]" term=3 +2024-07-05T13:35:44.977Z [WARN] server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34: rejecting vote request since we have a leader: from=a2dec55a-4a20-7ae4-41c7-054e74bb7526 leader=d651c130-76c7-27d2-aa7e-cb454000b9bc leader-id=server-d651c130-76c7-27d2-aa7e-cb454000b9bc +2024-07-05T13:35:44.991Z [WARN] server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34: heartbeat timeout reached, starting election: last-leader-addr=d651c130-76c7-27d2-aa7e-cb454000b9bc last-leader-id=server-d651c130-76c7-27d2-aa7e-cb454000b9bc +2024-07-05T13:35:44.991Z [INFO] server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34: entering candidate state: node="Node at 5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34 [Candidate]" term=3 +2024-07-05T13:35:44.992Z [INFO] server-a2dec55a-4a20-7ae4-41c7-054e74bb7526: duplicate requestVote for same term: term=3 +2024-07-05T13:35:45.043Z [WARN] server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34: Election timeout reached, restarting election +2024-07-05T13:35:45.043Z [INFO] server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34: entering candidate state: node="Node at 5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34 [Candidate]" term=4 +2024-07-05T13:35:45.044Z [INFO] server-a2dec55a-4a20-7ae4-41c7-054e74bb7526: entering follower state: follower="Node at a2dec55a-4a20-7ae4-41c7-054e74bb7526 [Follower]" leader-address= leader-id= +2024-07-05T13:35:45.044Z [INFO] server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34: election won: term=4 tally=2 +2024-07-05T13:35:45.044Z [INFO] server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34: entering leader state: leader="Node at 5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34 [Leader]" +2024-07-05T13:35:45.044Z [INFO] server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34: added peer, starting replication: peer=server-a2dec55a-4a20-7ae4-41c7-054e74bb7526 +2024-07-05T13:35:45.046Z [INFO] server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34: pipelining replication: peer="{Voter server-a2dec55a-4a20-7ae4-41c7-054e74bb7526 a2dec55a-4a20-7ae4-41c7-054e74bb7526}" +2024-07-05T13:35:45.107Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:45.213Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:45.107810341 +0000 UTC m=+11.103731792 (1 nodes), 461.408µs from start of poll, 500.312652ms from cluster start. Timeout at 2024-07-05 13:35:45.213578117 +0000 UTC m=+11.209499569, 105.767777ms after stability +2024-07-05T13:35:45.214Z [INFO] server-5e1c9f90-cd09-9dc1-c050-c6f0a8db4f34: aborting pipeline replication: peer="{Voter server-a2dec55a-4a20-7ae4-41c7-054e74bb7526 a2dec55a-4a20-7ae4-41c7-054e74bb7526}" +--- PASS: TestRaft_RemoveLeader (0.62s) === RUN TestRaft_RemoveLeader_NoShutdown -2024-07-05T13:29:29.890Z [INFO] server-88c307b3-c232-a642-2a82-69a3a4083e2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88c307b3-c232-a642-2a82-69a3a4083e2f Address:88c307b3-c232-a642-2a82-69a3a4083e2f} {Suffrage:Voter ID:server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158 Address:1bf8a6ae-1d28-b945-2aa7-cd72b8f19158} {Suffrage:Voter ID:server-35199186-57cd-a5f3-e1e4-9ac9a0686cfd Address:35199186-57cd-a5f3-e1e4-9ac9a0686cfd}]" -2024-07-05T13:29:29.890Z [INFO] server-88c307b3-c232-a642-2a82-69a3a4083e2f: entering follower state: follower="Node at 88c307b3-c232-a642-2a82-69a3a4083e2f [Follower]" leader-address= leader-id= -2024-07-05T13:29:29.890Z [INFO] server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88c307b3-c232-a642-2a82-69a3a4083e2f Address:88c307b3-c232-a642-2a82-69a3a4083e2f} {Suffrage:Voter ID:server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158 Address:1bf8a6ae-1d28-b945-2aa7-cd72b8f19158} {Suffrage:Voter ID:server-35199186-57cd-a5f3-e1e4-9ac9a0686cfd Address:35199186-57cd-a5f3-e1e4-9ac9a0686cfd}]" -2024-07-05T13:29:29.891Z [INFO] server-35199186-57cd-a5f3-e1e4-9ac9a0686cfd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88c307b3-c232-a642-2a82-69a3a4083e2f Address:88c307b3-c232-a642-2a82-69a3a4083e2f} {Suffrage:Voter ID:server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158 Address:1bf8a6ae-1d28-b945-2aa7-cd72b8f19158} {Suffrage:Voter ID:server-35199186-57cd-a5f3-e1e4-9ac9a0686cfd Address:35199186-57cd-a5f3-e1e4-9ac9a0686cfd}]" -2024-07-05T13:29:29.891Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:29:29.891Z [INFO] server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158: entering follower state: follower="Node at 1bf8a6ae-1d28-b945-2aa7-cd72b8f19158 [Follower]" leader-address= leader-id= -2024-07-05T13:29:29.891Z [INFO] server-35199186-57cd-a5f3-e1e4-9ac9a0686cfd: entering follower state: follower="Node at 35199186-57cd-a5f3-e1e4-9ac9a0686cfd [Follower]" leader-address= leader-id= -2024-07-05T13:29:29.956Z [WARN] server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:29.956Z [INFO] server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158: entering candidate state: node="Node at 1bf8a6ae-1d28-b945-2aa7-cd72b8f19158 [Candidate]" term=2 -2024-07-05T13:29:29.956Z [INFO] server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158: election won: term=2 tally=2 -2024-07-05T13:29:29.956Z [INFO] server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158: entering leader state: leader="Node at 1bf8a6ae-1d28-b945-2aa7-cd72b8f19158 [Leader]" -2024-07-05T13:29:29.956Z [INFO] server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158: added peer, starting replication: peer=server-88c307b3-c232-a642-2a82-69a3a4083e2f -2024-07-05T13:29:29.956Z [INFO] server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158: added peer, starting replication: peer=server-35199186-57cd-a5f3-e1e4-9ac9a0686cfd -2024-07-05T13:29:29.956Z [INFO] server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158: pipelining replication: peer="{Voter server-35199186-57cd-a5f3-e1e4-9ac9a0686cfd 35199186-57cd-a5f3-e1e4-9ac9a0686cfd}" -2024-07-05T13:29:29.957Z [INFO] server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158: pipelining replication: peer="{Voter server-88c307b3-c232-a642-2a82-69a3a4083e2f 88c307b3-c232-a642-2a82-69a3a4083e2f}" -2024-07-05T13:29:30.062Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:29:29.957259492 +0000 UTC m=+10.727170167 (2 nodes), 65.753773ms from start of poll, 66.757538ms from cluster start. Timeout at 2024-07-05 13:29:30.062447482 +0000 UTC m=+10.832358157, 105.18799ms after stability -2024-07-05T13:29:30.062Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:30.168Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:30.062635755 +0000 UTC m=+10.832546430 (1 nodes), 3.16µs from start of poll, 172.133801ms from cluster start. Timeout at 2024-07-05 13:29:30.16827133 +0000 UTC m=+10.938182005, 105.635575ms after stability -2024-07-05T13:29:30.183Z [INFO] server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158: updating configuration: command=RemoveServer server-id=server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158 server-addr= servers="[{Suffrage:Voter ID:server-88c307b3-c232-a642-2a82-69a3a4083e2f Address:88c307b3-c232-a642-2a82-69a3a4083e2f} {Suffrage:Voter ID:server-35199186-57cd-a5f3-e1e4-9ac9a0686cfd Address:35199186-57cd-a5f3-e1e4-9ac9a0686cfd}]" -2024-07-05T13:29:30.184Z [INFO] server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158: removed ourself, transitioning to follower -2024-07-05T13:29:30.184Z [INFO] server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158: entering follower state: follower="Node at 1bf8a6ae-1d28-b945-2aa7-cd72b8f19158 [Follower]" leader-address= leader-id= -2024-07-05T13:29:30.184Z [INFO] server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158: aborting pipeline replication: peer="{Voter server-35199186-57cd-a5f3-e1e4-9ac9a0686cfd 35199186-57cd-a5f3-e1e4-9ac9a0686cfd}" -2024-07-05T13:29:30.184Z [INFO] server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158: aborting pipeline replication: peer="{Voter server-88c307b3-c232-a642-2a82-69a3a4083e2f 88c307b3-c232-a642-2a82-69a3a4083e2f}" -2024-07-05T13:29:30.251Z [WARN] server-35199186-57cd-a5f3-e1e4-9ac9a0686cfd: heartbeat timeout reached, starting election: last-leader-addr=1bf8a6ae-1d28-b945-2aa7-cd72b8f19158 last-leader-id=server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158 -2024-07-05T13:29:30.251Z [INFO] server-35199186-57cd-a5f3-e1e4-9ac9a0686cfd: entering candidate state: node="Node at 35199186-57cd-a5f3-e1e4-9ac9a0686cfd [Candidate]" term=3 -2024-07-05T13:29:30.251Z [WARN] server-88c307b3-c232-a642-2a82-69a3a4083e2f: rejecting vote request since we have a leader: from=35199186-57cd-a5f3-e1e4-9ac9a0686cfd leader=1bf8a6ae-1d28-b945-2aa7-cd72b8f19158 leader-id=server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158 -2024-07-05T13:29:30.260Z [WARN] server-88c307b3-c232-a642-2a82-69a3a4083e2f: heartbeat timeout reached, starting election: last-leader-addr=1bf8a6ae-1d28-b945-2aa7-cd72b8f19158 last-leader-id=server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158 -2024-07-05T13:29:30.260Z [INFO] server-88c307b3-c232-a642-2a82-69a3a4083e2f: entering candidate state: node="Node at 88c307b3-c232-a642-2a82-69a3a4083e2f [Candidate]" term=3 -2024-07-05T13:29:30.260Z [INFO] server-35199186-57cd-a5f3-e1e4-9ac9a0686cfd: duplicate requestVote for same term: term=3 -2024-07-05T13:29:30.274Z [WARN] server-1bf8a6ae-1d28-b945-2aa7-cd72b8f19158: not part of stable configuration, aborting election -2024-07-05T13:29:30.290Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:30.326Z [WARN] server-35199186-57cd-a5f3-e1e4-9ac9a0686cfd: Election timeout reached, restarting election -2024-07-05T13:29:30.326Z [INFO] server-35199186-57cd-a5f3-e1e4-9ac9a0686cfd: entering candidate state: node="Node at 35199186-57cd-a5f3-e1e4-9ac9a0686cfd [Candidate]" term=4 -2024-07-05T13:29:30.327Z [INFO] server-88c307b3-c232-a642-2a82-69a3a4083e2f: entering follower state: follower="Node at 88c307b3-c232-a642-2a82-69a3a4083e2f [Follower]" leader-address= leader-id= -2024-07-05T13:29:30.327Z [INFO] server-35199186-57cd-a5f3-e1e4-9ac9a0686cfd: election won: term=4 tally=2 -2024-07-05T13:29:30.327Z [INFO] server-35199186-57cd-a5f3-e1e4-9ac9a0686cfd: entering leader state: leader="Node at 35199186-57cd-a5f3-e1e4-9ac9a0686cfd [Leader]" -2024-07-05T13:29:30.327Z [INFO] server-35199186-57cd-a5f3-e1e4-9ac9a0686cfd: added peer, starting replication: peer=server-88c307b3-c232-a642-2a82-69a3a4083e2f -2024-07-05T13:29:30.327Z [INFO] server-35199186-57cd-a5f3-e1e4-9ac9a0686cfd: pipelining replication: peer="{Voter server-88c307b3-c232-a642-2a82-69a3a4083e2f 88c307b3-c232-a642-2a82-69a3a4083e2f}" -2024-07-05T13:29:30.436Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:30.327397947 +0000 UTC m=+11.097308622 (1 nodes), 36.536278ms from start of poll, 436.895993ms from cluster start. Timeout at 2024-07-05 13:29:30.436196651 +0000 UTC m=+11.206107366, 108.798744ms after stability -2024-07-05T13:29:30.542Z [INFO] server-35199186-57cd-a5f3-e1e4-9ac9a0686cfd: aborting pipeline replication: peer="{Voter server-88c307b3-c232-a642-2a82-69a3a4083e2f 88c307b3-c232-a642-2a82-69a3a4083e2f}" ---- PASS: TestRaft_RemoveLeader_NoShutdown (0.65s) +2024-07-05T13:35:45.225Z [INFO] server-c821ecaf-bfe7-3d20-63eb-225ea289189d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c821ecaf-bfe7-3d20-63eb-225ea289189d Address:c821ecaf-bfe7-3d20-63eb-225ea289189d} {Suffrage:Voter ID:server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7 Address:2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7} {Suffrage:Voter ID:server-c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2 Address:c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2}]" +2024-07-05T13:35:45.225Z [INFO] server-c821ecaf-bfe7-3d20-63eb-225ea289189d: entering follower state: follower="Node at c821ecaf-bfe7-3d20-63eb-225ea289189d [Follower]" leader-address= leader-id= +2024-07-05T13:35:45.226Z [INFO] server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c821ecaf-bfe7-3d20-63eb-225ea289189d Address:c821ecaf-bfe7-3d20-63eb-225ea289189d} {Suffrage:Voter ID:server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7 Address:2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7} {Suffrage:Voter ID:server-c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2 Address:c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2}]" +2024-07-05T13:35:45.226Z [INFO] server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7: entering follower state: follower="Node at 2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7 [Follower]" leader-address= leader-id= +2024-07-05T13:35:45.226Z [INFO] server-c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c821ecaf-bfe7-3d20-63eb-225ea289189d Address:c821ecaf-bfe7-3d20-63eb-225ea289189d} {Suffrage:Voter ID:server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7 Address:2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7} {Suffrage:Voter ID:server-c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2 Address:c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2}]" +2024-07-05T13:35:45.227Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:35:45.227Z [INFO] server-c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2: entering follower state: follower="Node at c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2 [Follower]" leader-address= leader-id= +2024-07-05T13:35:45.298Z [WARN] server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:45.298Z [INFO] server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7: entering candidate state: node="Node at 2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7 [Candidate]" term=2 +2024-07-05T13:35:45.299Z [INFO] server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7: election won: term=2 tally=2 +2024-07-05T13:35:45.299Z [INFO] server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7: entering leader state: leader="Node at 2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7 [Leader]" +2024-07-05T13:35:45.299Z [INFO] server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7: added peer, starting replication: peer=server-c821ecaf-bfe7-3d20-63eb-225ea289189d +2024-07-05T13:35:45.299Z [INFO] server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7: added peer, starting replication: peer=server-c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2 +2024-07-05T13:35:45.300Z [INFO] server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7: pipelining replication: peer="{Voter server-c821ecaf-bfe7-3d20-63eb-225ea289189d c821ecaf-bfe7-3d20-63eb-225ea289189d}" +2024-07-05T13:35:45.300Z [INFO] server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7: pipelining replication: peer="{Voter server-c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2 c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2}" +2024-07-05T13:35:45.404Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:35:45.299307301 +0000 UTC m=+11.295229044 (2 nodes), 72.0806ms from start of poll, 74.886382ms from cluster start. Timeout at 2024-07-05 13:35:45.404628836 +0000 UTC m=+11.400550287, 105.321243ms after stability +2024-07-05T13:35:45.404Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:45.511Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:45.405084119 +0000 UTC m=+11.401005571 (1 nodes), 6.125µs from start of poll, 180.662909ms from cluster start. Timeout at 2024-07-05 13:35:45.511226972 +0000 UTC m=+11.507148423, 106.142852ms after stability +2024-07-05T13:35:45.542Z [INFO] server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7: updating configuration: command=RemoveServer server-id=server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7 server-addr= servers="[{Suffrage:Voter ID:server-c821ecaf-bfe7-3d20-63eb-225ea289189d Address:c821ecaf-bfe7-3d20-63eb-225ea289189d} {Suffrage:Voter ID:server-c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2 Address:c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2}]" +2024-07-05T13:35:45.544Z [INFO] server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7: removed ourself, transitioning to follower +2024-07-05T13:35:45.544Z [INFO] server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7: entering follower state: follower="Node at 2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7 [Follower]" leader-address= leader-id= +2024-07-05T13:35:45.544Z [INFO] server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7: aborting pipeline replication: peer="{Voter server-c821ecaf-bfe7-3d20-63eb-225ea289189d c821ecaf-bfe7-3d20-63eb-225ea289189d}" +2024-07-05T13:35:45.544Z [INFO] server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7: aborting pipeline replication: peer="{Voter server-c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2 c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2}" +2024-07-05T13:35:45.613Z [WARN] server-c821ecaf-bfe7-3d20-63eb-225ea289189d: heartbeat timeout reached, starting election: last-leader-addr=2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7 last-leader-id=server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7 +2024-07-05T13:35:45.613Z [INFO] server-c821ecaf-bfe7-3d20-63eb-225ea289189d: entering candidate state: node="Node at c821ecaf-bfe7-3d20-63eb-225ea289189d [Candidate]" term=3 +2024-07-05T13:35:45.613Z [WARN] server-c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2: rejecting vote request since we have a leader: from=c821ecaf-bfe7-3d20-63eb-225ea289189d leader=2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7 leader-id=server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7 +2024-07-05T13:35:45.621Z [WARN] server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7: not part of stable configuration, aborting election +2024-07-05T13:35:45.648Z [WARN] server-c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2: heartbeat timeout reached, starting election: last-leader-addr=2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7 last-leader-id=server-2531b0a4-3c7a-a00d-9d0d-9f44c886f1b7 +2024-07-05T13:35:45.648Z [INFO] server-c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2: entering candidate state: node="Node at c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2 [Candidate]" term=3 +2024-07-05T13:35:45.649Z [INFO] server-c821ecaf-bfe7-3d20-63eb-225ea289189d: duplicate requestVote for same term: term=3 +2024-07-05T13:35:45.651Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:45.673Z [WARN] server-c821ecaf-bfe7-3d20-63eb-225ea289189d: Election timeout reached, restarting election +2024-07-05T13:35:45.673Z [INFO] server-c821ecaf-bfe7-3d20-63eb-225ea289189d: entering candidate state: node="Node at c821ecaf-bfe7-3d20-63eb-225ea289189d [Candidate]" term=4 +2024-07-05T13:35:45.674Z [INFO] server-c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2: entering follower state: follower="Node at c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2 [Follower]" leader-address= leader-id= +2024-07-05T13:35:45.674Z [INFO] server-c821ecaf-bfe7-3d20-63eb-225ea289189d: election won: term=4 tally=2 +2024-07-05T13:35:45.674Z [INFO] server-c821ecaf-bfe7-3d20-63eb-225ea289189d: entering leader state: leader="Node at c821ecaf-bfe7-3d20-63eb-225ea289189d [Leader]" +2024-07-05T13:35:45.674Z [INFO] server-c821ecaf-bfe7-3d20-63eb-225ea289189d: added peer, starting replication: peer=server-c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2 +2024-07-05T13:35:45.675Z [INFO] server-c821ecaf-bfe7-3d20-63eb-225ea289189d: pipelining replication: peer="{Voter server-c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2 c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2}" +2024-07-05T13:35:45.780Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:45.674570446 +0000 UTC m=+11.670500648 (1 nodes), 22.553297ms from start of poll, 450.157986ms from cluster start. Timeout at 2024-07-05 13:35:45.780526636 +0000 UTC m=+11.776448087, 105.947439ms after stability +2024-07-05T13:35:45.886Z [INFO] server-c821ecaf-bfe7-3d20-63eb-225ea289189d: aborting pipeline replication: peer="{Voter server-c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2 c985fe42-ae3c-a47d-8b0a-7a81f51bb9a2}" +--- PASS: TestRaft_RemoveLeader_NoShutdown (0.67s) === RUN TestRaft_RemoveFollower_SplitCluster -2024-07-05T13:29:30.548Z [INFO] server-fc5a6625-b69f-865d-d1e5-2564d4394ffd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc5a6625-b69f-865d-d1e5-2564d4394ffd Address:fc5a6625-b69f-865d-d1e5-2564d4394ffd} {Suffrage:Voter ID:server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3 Address:ab91a0db-697d-b3bc-d7db-b77ec904bcd3} {Suffrage:Voter ID:server-e12db3f0-b69f-7b72-f3c2-a72e8c19642b Address:e12db3f0-b69f-7b72-f3c2-a72e8c19642b} {Suffrage:Voter ID:server-f6eddc5a-7aa0-396d-37fd-fb801e98dc30 Address:f6eddc5a-7aa0-396d-37fd-fb801e98dc30}]" -2024-07-05T13:29:30.548Z [INFO] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc5a6625-b69f-865d-d1e5-2564d4394ffd Address:fc5a6625-b69f-865d-d1e5-2564d4394ffd} {Suffrage:Voter ID:server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3 Address:ab91a0db-697d-b3bc-d7db-b77ec904bcd3} {Suffrage:Voter ID:server-e12db3f0-b69f-7b72-f3c2-a72e8c19642b Address:e12db3f0-b69f-7b72-f3c2-a72e8c19642b} {Suffrage:Voter ID:server-f6eddc5a-7aa0-396d-37fd-fb801e98dc30 Address:f6eddc5a-7aa0-396d-37fd-fb801e98dc30}]" -2024-07-05T13:29:30.548Z [INFO] server-e12db3f0-b69f-7b72-f3c2-a72e8c19642b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc5a6625-b69f-865d-d1e5-2564d4394ffd Address:fc5a6625-b69f-865d-d1e5-2564d4394ffd} {Suffrage:Voter ID:server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3 Address:ab91a0db-697d-b3bc-d7db-b77ec904bcd3} {Suffrage:Voter ID:server-e12db3f0-b69f-7b72-f3c2-a72e8c19642b Address:e12db3f0-b69f-7b72-f3c2-a72e8c19642b} {Suffrage:Voter ID:server-f6eddc5a-7aa0-396d-37fd-fb801e98dc30 Address:f6eddc5a-7aa0-396d-37fd-fb801e98dc30}]" -2024-07-05T13:29:30.548Z [INFO] server-f6eddc5a-7aa0-396d-37fd-fb801e98dc30: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc5a6625-b69f-865d-d1e5-2564d4394ffd Address:fc5a6625-b69f-865d-d1e5-2564d4394ffd} {Suffrage:Voter ID:server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3 Address:ab91a0db-697d-b3bc-d7db-b77ec904bcd3} {Suffrage:Voter ID:server-e12db3f0-b69f-7b72-f3c2-a72e8c19642b Address:e12db3f0-b69f-7b72-f3c2-a72e8c19642b} {Suffrage:Voter ID:server-f6eddc5a-7aa0-396d-37fd-fb801e98dc30 Address:f6eddc5a-7aa0-396d-37fd-fb801e98dc30}]" -2024-07-05T13:29:30.548Z [INFO] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: entering follower state: follower="Node at ab91a0db-697d-b3bc-d7db-b77ec904bcd3 [Follower]" leader-address= leader-id= -2024-07-05T13:29:30.549Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:30.549Z [INFO] server-fc5a6625-b69f-865d-d1e5-2564d4394ffd: entering follower state: follower="Node at fc5a6625-b69f-865d-d1e5-2564d4394ffd [Follower]" leader-address= leader-id= -2024-07-05T13:29:30.549Z [INFO] server-e12db3f0-b69f-7b72-f3c2-a72e8c19642b: entering follower state: follower="Node at e12db3f0-b69f-7b72-f3c2-a72e8c19642b [Follower]" leader-address= leader-id= -2024-07-05T13:29:30.549Z [INFO] server-f6eddc5a-7aa0-396d-37fd-fb801e98dc30: entering follower state: follower="Node at f6eddc5a-7aa0-396d-37fd-fb801e98dc30 [Follower]" leader-address= leader-id= -2024-07-05T13:29:30.612Z [WARN] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:30.612Z [INFO] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: entering candidate state: node="Node at ab91a0db-697d-b3bc-d7db-b77ec904bcd3 [Candidate]" term=2 -2024-07-05T13:29:30.612Z [INFO] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: election won: term=2 tally=3 -2024-07-05T13:29:30.612Z [INFO] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: entering leader state: leader="Node at ab91a0db-697d-b3bc-d7db-b77ec904bcd3 [Leader]" -2024-07-05T13:29:30.612Z [INFO] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: added peer, starting replication: peer=server-fc5a6625-b69f-865d-d1e5-2564d4394ffd -2024-07-05T13:29:30.612Z [INFO] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: added peer, starting replication: peer=server-e12db3f0-b69f-7b72-f3c2-a72e8c19642b -2024-07-05T13:29:30.612Z [INFO] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: added peer, starting replication: peer=server-f6eddc5a-7aa0-396d-37fd-fb801e98dc30 -2024-07-05T13:29:30.613Z [INFO] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: pipelining replication: peer="{Voter server-f6eddc5a-7aa0-396d-37fd-fb801e98dc30 f6eddc5a-7aa0-396d-37fd-fb801e98dc30}" -2024-07-05T13:29:30.613Z [INFO] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: pipelining replication: peer="{Voter server-fc5a6625-b69f-865d-d1e5-2564d4394ffd fc5a6625-b69f-865d-d1e5-2564d4394ffd}" -2024-07-05T13:29:30.613Z [INFO] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: pipelining replication: peer="{Voter server-e12db3f0-b69f-7b72-f3c2-a72e8c19642b e12db3f0-b69f-7b72-f3c2-a72e8c19642b}" -2024-07-05T13:29:30.719Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:30.613127484 +0000 UTC m=+11.383038159 (1 nodes), 64.065593ms from start of poll, 65.147595ms from cluster start. Timeout at 2024-07-05 13:29:30.71899177 +0000 UTC m=+11.488902445, 105.864286ms after stability -2024-07-05T13:29:30.825Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:29:30.931Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:29:30.825529193 +0000 UTC m=+11.595439868 (3 nodes), 2.52µs from start of poll, 277.549304ms from cluster start. Timeout at 2024-07-05 13:29:30.931316041 +0000 UTC m=+11.701226716, 105.786848ms after stability -2024-07-05T13:29:30.931Z [INFO] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: updating configuration: command=RemoveServer server-id=server-f6eddc5a-7aa0-396d-37fd-fb801e98dc30 server-addr= servers="[{Suffrage:Voter ID:server-fc5a6625-b69f-865d-d1e5-2564d4394ffd Address:fc5a6625-b69f-865d-d1e5-2564d4394ffd} {Suffrage:Voter ID:server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3 Address:ab91a0db-697d-b3bc-d7db-b77ec904bcd3} {Suffrage:Voter ID:server-e12db3f0-b69f-7b72-f3c2-a72e8c19642b Address:e12db3f0-b69f-7b72-f3c2-a72e8c19642b}]" -2024-07-05T13:29:30.931Z [INFO] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: removed peer, stopping replication: peer=server-f6eddc5a-7aa0-396d-37fd-fb801e98dc30 last-index=3 -2024-07-05T13:29:30.931Z [INFO] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: aborting pipeline replication: peer="{Voter server-f6eddc5a-7aa0-396d-37fd-fb801e98dc30 f6eddc5a-7aa0-396d-37fd-fb801e98dc30}" -2024-07-05T13:29:30.931Z [ERROR] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: failed to pipeline appendEntries: peer="{Voter server-fc5a6625-b69f-865d-d1e5-2564d4394ffd fc5a6625-b69f-865d-d1e5-2564d4394ffd}" error="append pipeline closed" -2024-07-05T13:29:30.931Z [INFO] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: aborting pipeline replication: peer="{Voter server-fc5a6625-b69f-865d-d1e5-2564d4394ffd fc5a6625-b69f-865d-d1e5-2564d4394ffd}" -2024-07-05T13:29:30.931Z [ERROR] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: failed to pipeline appendEntries: peer="{Voter server-e12db3f0-b69f-7b72-f3c2-a72e8c19642b e12db3f0-b69f-7b72-f3c2-a72e8c19642b}" error="append pipeline closed" -2024-07-05T13:29:30.931Z [INFO] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: aborting pipeline replication: peer="{Voter server-e12db3f0-b69f-7b72-f3c2-a72e8c19642b e12db3f0-b69f-7b72-f3c2-a72e8c19642b}" -2024-07-05T13:29:30.933Z [ERROR] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: failed to heartbeat to: peer=e12db3f0-b69f-7b72-f3c2-a72e8c19642b backoff time=10ms error="failed to connect to peer: e12db3f0-b69f-7b72-f3c2-a72e8c19642b" -2024-07-05T13:29:30.938Z [ERROR] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: failed to heartbeat to: peer=fc5a6625-b69f-865d-d1e5-2564d4394ffd backoff time=10ms error="failed to connect to peer: fc5a6625-b69f-865d-d1e5-2564d4394ffd" -2024-07-05T13:29:30.940Z [ERROR] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: failed to appendEntries to: peer="{Voter server-e12db3f0-b69f-7b72-f3c2-a72e8c19642b e12db3f0-b69f-7b72-f3c2-a72e8c19642b}" error="failed to connect to peer: e12db3f0-b69f-7b72-f3c2-a72e8c19642b" -2024-07-05T13:29:30.940Z [ERROR] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: failed to appendEntries to: peer="{Voter server-fc5a6625-b69f-865d-d1e5-2564d4394ffd fc5a6625-b69f-865d-d1e5-2564d4394ffd}" error="failed to connect to peer: fc5a6625-b69f-865d-d1e5-2564d4394ffd" -2024-07-05T13:29:30.953Z [ERROR] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: failed to heartbeat to: peer=e12db3f0-b69f-7b72-f3c2-a72e8c19642b backoff time=10ms error="failed to connect to peer: e12db3f0-b69f-7b72-f3c2-a72e8c19642b" -2024-07-05T13:29:30.955Z [ERROR] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: failed to heartbeat to: peer=fc5a6625-b69f-865d-d1e5-2564d4394ffd backoff time=10ms error="failed to connect to peer: fc5a6625-b69f-865d-d1e5-2564d4394ffd" -2024-07-05T13:29:30.959Z [ERROR] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: failed to appendEntries to: peer="{Voter server-fc5a6625-b69f-865d-d1e5-2564d4394ffd fc5a6625-b69f-865d-d1e5-2564d4394ffd}" error="failed to connect to peer: fc5a6625-b69f-865d-d1e5-2564d4394ffd" -2024-07-05T13:29:30.960Z [ERROR] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: failed to appendEntries to: peer="{Voter server-e12db3f0-b69f-7b72-f3c2-a72e8c19642b e12db3f0-b69f-7b72-f3c2-a72e8c19642b}" error="failed to connect to peer: e12db3f0-b69f-7b72-f3c2-a72e8c19642b" -2024-07-05T13:29:30.973Z [ERROR] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: failed to heartbeat to: peer=e12db3f0-b69f-7b72-f3c2-a72e8c19642b backoff time=10ms error="failed to connect to peer: e12db3f0-b69f-7b72-f3c2-a72e8c19642b" -2024-07-05T13:29:30.976Z [ERROR] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: failed to heartbeat to: peer=fc5a6625-b69f-865d-d1e5-2564d4394ffd backoff time=10ms error="failed to connect to peer: fc5a6625-b69f-865d-d1e5-2564d4394ffd" -2024-07-05T13:29:30.977Z [ERROR] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: failed to appendEntries to: peer="{Voter server-fc5a6625-b69f-865d-d1e5-2564d4394ffd fc5a6625-b69f-865d-d1e5-2564d4394ffd}" error="failed to connect to peer: fc5a6625-b69f-865d-d1e5-2564d4394ffd" -2024-07-05T13:29:30.978Z [WARN] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: failed to contact: server-id=server-e12db3f0-b69f-7b72-f3c2-a72e8c19642b time=51.293961ms -2024-07-05T13:29:30.978Z [ERROR] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: failed to appendEntries to: peer="{Voter server-e12db3f0-b69f-7b72-f3c2-a72e8c19642b e12db3f0-b69f-7b72-f3c2-a72e8c19642b}" error="failed to connect to peer: e12db3f0-b69f-7b72-f3c2-a72e8c19642b" -2024-07-05T13:29:30.988Z [WARN] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: failed to contact: server-id=server-fc5a6625-b69f-865d-d1e5-2564d4394ffd time=57.770733ms -2024-07-05T13:29:30.988Z [WARN] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: failed to contact: server-id=server-e12db3f0-b69f-7b72-f3c2-a72e8c19642b time=61.437365ms -2024-07-05T13:29:30.988Z [WARN] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: failed to contact quorum of nodes, stepping down -2024-07-05T13:29:30.988Z [INFO] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: entering follower state: follower="Node at ab91a0db-697d-b3bc-d7db-b77ec904bcd3 [Follower]" leader-address= leader-id= -2024-07-05T13:29:30.988Z [ERROR] server-ab91a0db-697d-b3bc-d7db-b77ec904bcd3: failed to appendEntries to: peer="{Voter server-fc5a6625-b69f-865d-d1e5-2564d4394ffd fc5a6625-b69f-865d-d1e5-2564d4394ffd}" error="failed to connect to peer: fc5a6625-b69f-865d-d1e5-2564d4394ffd" ---- PASS: TestRaft_RemoveFollower_SplitCluster (0.45s) +2024-07-05T13:35:45.898Z [INFO] server-7466ca4e-aac5-1454-062b-385374411e03: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7466ca4e-aac5-1454-062b-385374411e03 Address:7466ca4e-aac5-1454-062b-385374411e03} {Suffrage:Voter ID:server-3e384d23-d9a5-e25b-6b0f-acd9f5c31b55 Address:3e384d23-d9a5-e25b-6b0f-acd9f5c31b55} {Suffrage:Voter ID:server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599 Address:8a9d1f8f-6189-6c27-0ff6-78cfe8cec599} {Suffrage:Voter ID:server-dbc76036-0333-4dad-aece-76e662cab905 Address:dbc76036-0333-4dad-aece-76e662cab905}]" +2024-07-05T13:35:45.899Z [INFO] server-7466ca4e-aac5-1454-062b-385374411e03: entering follower state: follower="Node at 7466ca4e-aac5-1454-062b-385374411e03 [Follower]" leader-address= leader-id= +2024-07-05T13:35:45.899Z [INFO] server-3e384d23-d9a5-e25b-6b0f-acd9f5c31b55: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7466ca4e-aac5-1454-062b-385374411e03 Address:7466ca4e-aac5-1454-062b-385374411e03} {Suffrage:Voter ID:server-3e384d23-d9a5-e25b-6b0f-acd9f5c31b55 Address:3e384d23-d9a5-e25b-6b0f-acd9f5c31b55} {Suffrage:Voter ID:server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599 Address:8a9d1f8f-6189-6c27-0ff6-78cfe8cec599} {Suffrage:Voter ID:server-dbc76036-0333-4dad-aece-76e662cab905 Address:dbc76036-0333-4dad-aece-76e662cab905}]" +2024-07-05T13:35:45.899Z [INFO] server-3e384d23-d9a5-e25b-6b0f-acd9f5c31b55: entering follower state: follower="Node at 3e384d23-d9a5-e25b-6b0f-acd9f5c31b55 [Follower]" leader-address= leader-id= +2024-07-05T13:35:45.900Z [INFO] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7466ca4e-aac5-1454-062b-385374411e03 Address:7466ca4e-aac5-1454-062b-385374411e03} {Suffrage:Voter ID:server-3e384d23-d9a5-e25b-6b0f-acd9f5c31b55 Address:3e384d23-d9a5-e25b-6b0f-acd9f5c31b55} {Suffrage:Voter ID:server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599 Address:8a9d1f8f-6189-6c27-0ff6-78cfe8cec599} {Suffrage:Voter ID:server-dbc76036-0333-4dad-aece-76e662cab905 Address:dbc76036-0333-4dad-aece-76e662cab905}]" +2024-07-05T13:35:45.900Z [INFO] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: entering follower state: follower="Node at 8a9d1f8f-6189-6c27-0ff6-78cfe8cec599 [Follower]" leader-address= leader-id= +2024-07-05T13:35:45.901Z [INFO] server-dbc76036-0333-4dad-aece-76e662cab905: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7466ca4e-aac5-1454-062b-385374411e03 Address:7466ca4e-aac5-1454-062b-385374411e03} {Suffrage:Voter ID:server-3e384d23-d9a5-e25b-6b0f-acd9f5c31b55 Address:3e384d23-d9a5-e25b-6b0f-acd9f5c31b55} {Suffrage:Voter ID:server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599 Address:8a9d1f8f-6189-6c27-0ff6-78cfe8cec599} {Suffrage:Voter ID:server-dbc76036-0333-4dad-aece-76e662cab905 Address:dbc76036-0333-4dad-aece-76e662cab905}]" +2024-07-05T13:35:45.901Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:45.902Z [INFO] server-dbc76036-0333-4dad-aece-76e662cab905: entering follower state: follower="Node at dbc76036-0333-4dad-aece-76e662cab905 [Follower]" leader-address= leader-id= +2024-07-05T13:35:45.960Z [WARN] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:45.960Z [INFO] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: entering candidate state: node="Node at 8a9d1f8f-6189-6c27-0ff6-78cfe8cec599 [Candidate]" term=2 +2024-07-05T13:35:45.961Z [INFO] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: election won: term=2 tally=3 +2024-07-05T13:35:45.961Z [INFO] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: entering leader state: leader="Node at 8a9d1f8f-6189-6c27-0ff6-78cfe8cec599 [Leader]" +2024-07-05T13:35:45.962Z [INFO] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: added peer, starting replication: peer=server-7466ca4e-aac5-1454-062b-385374411e03 +2024-07-05T13:35:45.962Z [INFO] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: added peer, starting replication: peer=server-3e384d23-d9a5-e25b-6b0f-acd9f5c31b55 +2024-07-05T13:35:45.962Z [INFO] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: added peer, starting replication: peer=server-dbc76036-0333-4dad-aece-76e662cab905 +2024-07-05T13:35:45.962Z [INFO] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: pipelining replication: peer="{Voter server-7466ca4e-aac5-1454-062b-385374411e03 7466ca4e-aac5-1454-062b-385374411e03}" +2024-07-05T13:35:45.963Z [INFO] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: pipelining replication: peer="{Voter server-dbc76036-0333-4dad-aece-76e662cab905 dbc76036-0333-4dad-aece-76e662cab905}" +2024-07-05T13:35:45.963Z [INFO] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: pipelining replication: peer="{Voter server-3e384d23-d9a5-e25b-6b0f-acd9f5c31b55 3e384d23-d9a5-e25b-6b0f-acd9f5c31b55}" +2024-07-05T13:35:46.067Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:45.961899448 +0000 UTC m=+11.957820899 (1 nodes), 59.938738ms from start of poll, 63.951415ms from cluster start. Timeout at 2024-07-05 13:35:46.067242565 +0000 UTC m=+12.063164017, 105.343118ms after stability +2024-07-05T13:35:46.172Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:35:46.278Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:35:46.172941509 +0000 UTC m=+12.168862961 (3 nodes), 10.5µs from start of poll, 274.993477ms from cluster start. Timeout at 2024-07-05 13:35:46.277971966 +0000 UTC m=+12.273893417, 105.030456ms after stability +2024-07-05T13:35:46.278Z [ERROR] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: failed to heartbeat to: peer=3e384d23-d9a5-e25b-6b0f-acd9f5c31b55 backoff time=10ms error="failed to connect to peer: 3e384d23-d9a5-e25b-6b0f-acd9f5c31b55" +2024-07-05T13:35:46.278Z [INFO] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: updating configuration: command=RemoveServer server-id=server-dbc76036-0333-4dad-aece-76e662cab905 server-addr= servers="[{Suffrage:Voter ID:server-7466ca4e-aac5-1454-062b-385374411e03 Address:7466ca4e-aac5-1454-062b-385374411e03} {Suffrage:Voter ID:server-3e384d23-d9a5-e25b-6b0f-acd9f5c31b55 Address:3e384d23-d9a5-e25b-6b0f-acd9f5c31b55} {Suffrage:Voter ID:server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599 Address:8a9d1f8f-6189-6c27-0ff6-78cfe8cec599}]" +2024-07-05T13:35:46.278Z [ERROR] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: failed to pipeline appendEntries: peer="{Voter server-7466ca4e-aac5-1454-062b-385374411e03 7466ca4e-aac5-1454-062b-385374411e03}" error="append pipeline closed" +2024-07-05T13:35:46.279Z [INFO] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: removed peer, stopping replication: peer=server-dbc76036-0333-4dad-aece-76e662cab905 last-index=3 +2024-07-05T13:35:46.279Z [ERROR] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: failed to pipeline appendEntries: peer="{Voter server-3e384d23-d9a5-e25b-6b0f-acd9f5c31b55 3e384d23-d9a5-e25b-6b0f-acd9f5c31b55}" error="append pipeline closed" +2024-07-05T13:35:46.279Z [INFO] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: aborting pipeline replication: peer="{Voter server-dbc76036-0333-4dad-aece-76e662cab905 dbc76036-0333-4dad-aece-76e662cab905}" +2024-07-05T13:35:46.279Z [INFO] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: aborting pipeline replication: peer="{Voter server-3e384d23-d9a5-e25b-6b0f-acd9f5c31b55 3e384d23-d9a5-e25b-6b0f-acd9f5c31b55}" +2024-07-05T13:35:46.279Z [INFO] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: aborting pipeline replication: peer="{Voter server-7466ca4e-aac5-1454-062b-385374411e03 7466ca4e-aac5-1454-062b-385374411e03}" +2024-07-05T13:35:46.279Z [ERROR] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: failed to appendEntries to: peer="{Voter server-7466ca4e-aac5-1454-062b-385374411e03 7466ca4e-aac5-1454-062b-385374411e03}" error="failed to connect to peer: 7466ca4e-aac5-1454-062b-385374411e03" +2024-07-05T13:35:46.281Z [ERROR] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: failed to heartbeat to: peer=7466ca4e-aac5-1454-062b-385374411e03 backoff time=10ms error="failed to connect to peer: 7466ca4e-aac5-1454-062b-385374411e03" +2024-07-05T13:35:46.287Z [ERROR] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: failed to appendEntries to: peer="{Voter server-3e384d23-d9a5-e25b-6b0f-acd9f5c31b55 3e384d23-d9a5-e25b-6b0f-acd9f5c31b55}" error="failed to connect to peer: 3e384d23-d9a5-e25b-6b0f-acd9f5c31b55" +2024-07-05T13:35:46.294Z [ERROR] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: failed to heartbeat to: peer=3e384d23-d9a5-e25b-6b0f-acd9f5c31b55 backoff time=10ms error="failed to connect to peer: 3e384d23-d9a5-e25b-6b0f-acd9f5c31b55" +2024-07-05T13:35:46.297Z [ERROR] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: failed to appendEntries to: peer="{Voter server-7466ca4e-aac5-1454-062b-385374411e03 7466ca4e-aac5-1454-062b-385374411e03}" error="failed to connect to peer: 7466ca4e-aac5-1454-062b-385374411e03" +2024-07-05T13:35:46.300Z [ERROR] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: failed to heartbeat to: peer=7466ca4e-aac5-1454-062b-385374411e03 backoff time=10ms error="failed to connect to peer: 7466ca4e-aac5-1454-062b-385374411e03" +2024-07-05T13:35:46.307Z [ERROR] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: failed to appendEntries to: peer="{Voter server-3e384d23-d9a5-e25b-6b0f-acd9f5c31b55 3e384d23-d9a5-e25b-6b0f-acd9f5c31b55}" error="failed to connect to peer: 3e384d23-d9a5-e25b-6b0f-acd9f5c31b55" +2024-07-05T13:35:46.315Z [ERROR] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: failed to heartbeat to: peer=3e384d23-d9a5-e25b-6b0f-acd9f5c31b55 backoff time=10ms error="failed to connect to peer: 3e384d23-d9a5-e25b-6b0f-acd9f5c31b55" +2024-07-05T13:35:46.315Z [ERROR] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: failed to appendEntries to: peer="{Voter server-7466ca4e-aac5-1454-062b-385374411e03 7466ca4e-aac5-1454-062b-385374411e03}" error="failed to connect to peer: 7466ca4e-aac5-1454-062b-385374411e03" +2024-07-05T13:35:46.322Z [ERROR] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: failed to heartbeat to: peer=7466ca4e-aac5-1454-062b-385374411e03 backoff time=10ms error="failed to connect to peer: 7466ca4e-aac5-1454-062b-385374411e03" +2024-07-05T13:35:46.325Z [ERROR] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: failed to appendEntries to: peer="{Voter server-3e384d23-d9a5-e25b-6b0f-acd9f5c31b55 3e384d23-d9a5-e25b-6b0f-acd9f5c31b55}" error="failed to connect to peer: 3e384d23-d9a5-e25b-6b0f-acd9f5c31b55" +2024-07-05T13:35:46.325Z [WARN] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: failed to contact: server-id=server-7466ca4e-aac5-1454-062b-385374411e03 time=50.719615ms +2024-07-05T13:35:46.325Z [WARN] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: failed to contact: server-id=server-3e384d23-d9a5-e25b-6b0f-acd9f5c31b55 time=50.273665ms +2024-07-05T13:35:46.326Z [WARN] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: failed to contact quorum of nodes, stepping down +2024-07-05T13:35:46.326Z [INFO] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: entering follower state: follower="Node at 8a9d1f8f-6189-6c27-0ff6-78cfe8cec599 [Follower]" leader-address= leader-id= +2024-07-05T13:35:46.326Z [ERROR] server-8a9d1f8f-6189-6c27-0ff6-78cfe8cec599: failed to appendEntries to: peer="{Voter server-7466ca4e-aac5-1454-062b-385374411e03 7466ca4e-aac5-1454-062b-385374411e03}" error="failed to connect to peer: 7466ca4e-aac5-1454-062b-385374411e03" +--- PASS: TestRaft_RemoveFollower_SplitCluster (0.44s) === RUN TestRaft_AddKnownPeer -2024-07-05T13:29:30.991Z [INFO] server-b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b Address:b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b} {Suffrage:Voter ID:server-1ff239c3-7ea6-9f63-957e-3deab8589514 Address:1ff239c3-7ea6-9f63-957e-3deab8589514} {Suffrage:Voter ID:server-08bfca71-7ea8-704b-9366-37869c40371d Address:08bfca71-7ea8-704b-9366-37869c40371d}]" -2024-07-05T13:29:30.991Z [INFO] server-1ff239c3-7ea6-9f63-957e-3deab8589514: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b Address:b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b} {Suffrage:Voter ID:server-1ff239c3-7ea6-9f63-957e-3deab8589514 Address:1ff239c3-7ea6-9f63-957e-3deab8589514} {Suffrage:Voter ID:server-08bfca71-7ea8-704b-9366-37869c40371d Address:08bfca71-7ea8-704b-9366-37869c40371d}]" -2024-07-05T13:29:30.991Z [INFO] server-08bfca71-7ea8-704b-9366-37869c40371d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b Address:b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b} {Suffrage:Voter ID:server-1ff239c3-7ea6-9f63-957e-3deab8589514 Address:1ff239c3-7ea6-9f63-957e-3deab8589514} {Suffrage:Voter ID:server-08bfca71-7ea8-704b-9366-37869c40371d Address:08bfca71-7ea8-704b-9366-37869c40371d}]" -2024-07-05T13:29:30.991Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:30.991Z [INFO] server-b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b: entering follower state: follower="Node at b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b [Follower]" leader-address= leader-id= -2024-07-05T13:29:30.991Z [INFO] server-1ff239c3-7ea6-9f63-957e-3deab8589514: entering follower state: follower="Node at 1ff239c3-7ea6-9f63-957e-3deab8589514 [Follower]" leader-address= leader-id= -2024-07-05T13:29:30.991Z [INFO] server-08bfca71-7ea8-704b-9366-37869c40371d: entering follower state: follower="Node at 08bfca71-7ea8-704b-9366-37869c40371d [Follower]" leader-address= leader-id= -2024-07-05T13:29:31.058Z [WARN] server-b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:31.058Z [INFO] server-b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b: entering candidate state: node="Node at b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b [Candidate]" term=2 -2024-07-05T13:29:31.058Z [INFO] server-b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b: election won: term=2 tally=2 -2024-07-05T13:29:31.058Z [INFO] server-b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b: entering leader state: leader="Node at b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b [Leader]" -2024-07-05T13:29:31.058Z [INFO] server-b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b: added peer, starting replication: peer=server-1ff239c3-7ea6-9f63-957e-3deab8589514 -2024-07-05T13:29:31.058Z [INFO] server-b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b: added peer, starting replication: peer=server-08bfca71-7ea8-704b-9366-37869c40371d -2024-07-05T13:29:31.058Z [INFO] server-b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b: pipelining replication: peer="{Voter server-08bfca71-7ea8-704b-9366-37869c40371d 08bfca71-7ea8-704b-9366-37869c40371d}" -2024-07-05T13:29:31.059Z [INFO] server-b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b: pipelining replication: peer="{Voter server-1ff239c3-7ea6-9f63-957e-3deab8589514 1ff239c3-7ea6-9f63-957e-3deab8589514}" -2024-07-05T13:29:31.165Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:31.059186836 +0000 UTC m=+11.829097511 (1 nodes), 67.476553ms from start of poll, 68.210047ms from cluster start. Timeout at 2024-07-05 13:29:31.165418829 +0000 UTC m=+11.935329504, 106.231993ms after stability -2024-07-05T13:29:31.165Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:29:31.270Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:29:31.165567184 +0000 UTC m=+11.935477859 (2 nodes), 2.12µs from start of poll, 174.590395ms from cluster start. Timeout at 2024-07-05 13:29:31.2705759 +0000 UTC m=+12.040486575, 105.008716ms after stability -2024-07-05T13:29:31.270Z [INFO] server-b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b: updating configuration: command=AddVoter server-id=server-1ff239c3-7ea6-9f63-957e-3deab8589514 server-addr=1ff239c3-7ea6-9f63-957e-3deab8589514 servers="[{Suffrage:Voter ID:server-b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b Address:b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b} {Suffrage:Voter ID:server-1ff239c3-7ea6-9f63-957e-3deab8589514 Address:1ff239c3-7ea6-9f63-957e-3deab8589514} {Suffrage:Voter ID:server-08bfca71-7ea8-704b-9366-37869c40371d Address:08bfca71-7ea8-704b-9366-37869c40371d}]" -2024-07-05T13:29:31.271Z [INFO] server-b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b: aborting pipeline replication: peer="{Voter server-08bfca71-7ea8-704b-9366-37869c40371d 08bfca71-7ea8-704b-9366-37869c40371d}" -2024-07-05T13:29:31.271Z [INFO] server-b2d1e9e9-7e95-b3c6-9ae3-68d9d11c7e8b: aborting pipeline replication: peer="{Voter server-1ff239c3-7ea6-9f63-957e-3deab8589514 1ff239c3-7ea6-9f63-957e-3deab8589514}" +2024-07-05T13:35:46.338Z [INFO] server-98681a52-6672-a344-4a4b-c9869ee4bc0a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98681a52-6672-a344-4a4b-c9869ee4bc0a Address:98681a52-6672-a344-4a4b-c9869ee4bc0a} {Suffrage:Voter ID:server-9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8 Address:9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8} {Suffrage:Voter ID:server-73d446be-f73e-3060-113b-3903c06bbfab Address:73d446be-f73e-3060-113b-3903c06bbfab}]" +2024-07-05T13:35:46.338Z [INFO] server-98681a52-6672-a344-4a4b-c9869ee4bc0a: entering follower state: follower="Node at 98681a52-6672-a344-4a4b-c9869ee4bc0a [Follower]" leader-address= leader-id= +2024-07-05T13:35:46.339Z [INFO] server-9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98681a52-6672-a344-4a4b-c9869ee4bc0a Address:98681a52-6672-a344-4a4b-c9869ee4bc0a} {Suffrage:Voter ID:server-9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8 Address:9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8} {Suffrage:Voter ID:server-73d446be-f73e-3060-113b-3903c06bbfab Address:73d446be-f73e-3060-113b-3903c06bbfab}]" +2024-07-05T13:35:46.339Z [INFO] server-9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8: entering follower state: follower="Node at 9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8 [Follower]" leader-address= leader-id= +2024-07-05T13:35:46.340Z [INFO] server-73d446be-f73e-3060-113b-3903c06bbfab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98681a52-6672-a344-4a4b-c9869ee4bc0a Address:98681a52-6672-a344-4a4b-c9869ee4bc0a} {Suffrage:Voter ID:server-9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8 Address:9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8} {Suffrage:Voter ID:server-73d446be-f73e-3060-113b-3903c06bbfab Address:73d446be-f73e-3060-113b-3903c06bbfab}]" +2024-07-05T13:35:46.340Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:46.340Z [INFO] server-73d446be-f73e-3060-113b-3903c06bbfab: entering follower state: follower="Node at 73d446be-f73e-3060-113b-3903c06bbfab [Follower]" leader-address= leader-id= +2024-07-05T13:35:46.396Z [WARN] server-9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:46.397Z [INFO] server-9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8: entering candidate state: node="Node at 9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8 [Candidate]" term=2 +2024-07-05T13:35:46.397Z [INFO] server-9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8: election won: term=2 tally=2 +2024-07-05T13:35:46.397Z [INFO] server-9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8: entering leader state: leader="Node at 9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8 [Leader]" +2024-07-05T13:35:46.398Z [INFO] server-9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8: added peer, starting replication: peer=server-98681a52-6672-a344-4a4b-c9869ee4bc0a +2024-07-05T13:35:46.398Z [INFO] server-9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8: added peer, starting replication: peer=server-73d446be-f73e-3060-113b-3903c06bbfab +2024-07-05T13:35:46.398Z [INFO] server-9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8: pipelining replication: peer="{Voter server-98681a52-6672-a344-4a4b-c9869ee4bc0a 98681a52-6672-a344-4a4b-c9869ee4bc0a}" +2024-07-05T13:35:46.399Z [INFO] server-9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8: pipelining replication: peer="{Voter server-73d446be-f73e-3060-113b-3903c06bbfab 73d446be-f73e-3060-113b-3903c06bbfab}" +2024-07-05T13:35:46.503Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:46.397947441 +0000 UTC m=+12.393868893 (1 nodes), 57.226288ms from start of poll, 60.116069ms from cluster start. Timeout at 2024-07-05 13:35:46.503188185 +0000 UTC m=+12.499109637, 105.240744ms after stability +2024-07-05T13:35:46.503Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:35:46.609Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:35:46.503581345 +0000 UTC m=+12.499502505 (2 nodes), 7.875µs from start of poll, 165.749681ms from cluster start. Timeout at 2024-07-05 13:35:46.609236248 +0000 UTC m=+12.605157700, 105.655195ms after stability +2024-07-05T13:35:46.609Z [INFO] server-9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8: updating configuration: command=AddVoter server-id=server-98681a52-6672-a344-4a4b-c9869ee4bc0a server-addr=98681a52-6672-a344-4a4b-c9869ee4bc0a servers="[{Suffrage:Voter ID:server-98681a52-6672-a344-4a4b-c9869ee4bc0a Address:98681a52-6672-a344-4a4b-c9869ee4bc0a} {Suffrage:Voter ID:server-9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8 Address:9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8} {Suffrage:Voter ID:server-73d446be-f73e-3060-113b-3903c06bbfab Address:73d446be-f73e-3060-113b-3903c06bbfab}]" +2024-07-05T13:35:46.611Z [INFO] server-9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8: aborting pipeline replication: peer="{Voter server-98681a52-6672-a344-4a4b-c9869ee4bc0a 98681a52-6672-a344-4a4b-c9869ee4bc0a}" +2024-07-05T13:35:46.611Z [INFO] server-9d8d63ae-fa06-bb9e-4b4d-fe8fdcbb85f8: aborting pipeline replication: peer="{Voter server-73d446be-f73e-3060-113b-3903c06bbfab 73d446be-f73e-3060-113b-3903c06bbfab}" --- PASS: TestRaft_AddKnownPeer (0.28s) === RUN TestRaft_RemoveUnknownPeer -2024-07-05T13:29:31.273Z [INFO] server-3ab3d722-f0cf-bdbc-4734-416d6c849e90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ab3d722-f0cf-bdbc-4734-416d6c849e90 Address:3ab3d722-f0cf-bdbc-4734-416d6c849e90} {Suffrage:Voter ID:server-263bd370-a919-daf6-6105-bf3591640561 Address:263bd370-a919-daf6-6105-bf3591640561} {Suffrage:Voter ID:server-a32be3af-6d78-4033-c0b4-57ea88399745 Address:a32be3af-6d78-4033-c0b4-57ea88399745}]" -2024-07-05T13:29:31.274Z [INFO] server-263bd370-a919-daf6-6105-bf3591640561: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ab3d722-f0cf-bdbc-4734-416d6c849e90 Address:3ab3d722-f0cf-bdbc-4734-416d6c849e90} {Suffrage:Voter ID:server-263bd370-a919-daf6-6105-bf3591640561 Address:263bd370-a919-daf6-6105-bf3591640561} {Suffrage:Voter ID:server-a32be3af-6d78-4033-c0b4-57ea88399745 Address:a32be3af-6d78-4033-c0b4-57ea88399745}]" -2024-07-05T13:29:31.274Z [INFO] server-a32be3af-6d78-4033-c0b4-57ea88399745: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ab3d722-f0cf-bdbc-4734-416d6c849e90 Address:3ab3d722-f0cf-bdbc-4734-416d6c849e90} {Suffrage:Voter ID:server-263bd370-a919-daf6-6105-bf3591640561 Address:263bd370-a919-daf6-6105-bf3591640561} {Suffrage:Voter ID:server-a32be3af-6d78-4033-c0b4-57ea88399745 Address:a32be3af-6d78-4033-c0b4-57ea88399745}]" -2024-07-05T13:29:31.274Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:31.274Z [INFO] server-3ab3d722-f0cf-bdbc-4734-416d6c849e90: entering follower state: follower="Node at 3ab3d722-f0cf-bdbc-4734-416d6c849e90 [Follower]" leader-address= leader-id= -2024-07-05T13:29:31.274Z [INFO] server-263bd370-a919-daf6-6105-bf3591640561: entering follower state: follower="Node at 263bd370-a919-daf6-6105-bf3591640561 [Follower]" leader-address= leader-id= -2024-07-05T13:29:31.274Z [INFO] server-a32be3af-6d78-4033-c0b4-57ea88399745: entering follower state: follower="Node at a32be3af-6d78-4033-c0b4-57ea88399745 [Follower]" leader-address= leader-id= -2024-07-05T13:29:31.344Z [WARN] server-a32be3af-6d78-4033-c0b4-57ea88399745: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:31.344Z [WARN] server-3ab3d722-f0cf-bdbc-4734-416d6c849e90: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:31.344Z [INFO] server-a32be3af-6d78-4033-c0b4-57ea88399745: entering candidate state: node="Node at a32be3af-6d78-4033-c0b4-57ea88399745 [Candidate]" term=2 -2024-07-05T13:29:31.344Z [INFO] server-3ab3d722-f0cf-bdbc-4734-416d6c849e90: entering candidate state: node="Node at 3ab3d722-f0cf-bdbc-4734-416d6c849e90 [Candidate]" term=2 -2024-07-05T13:29:31.344Z [INFO] server-a32be3af-6d78-4033-c0b4-57ea88399745: duplicate requestVote for same term: term=2 -2024-07-05T13:29:31.344Z [INFO] server-a32be3af-6d78-4033-c0b4-57ea88399745: election won: term=2 tally=2 -2024-07-05T13:29:31.344Z [INFO] server-a32be3af-6d78-4033-c0b4-57ea88399745: entering leader state: leader="Node at a32be3af-6d78-4033-c0b4-57ea88399745 [Leader]" -2024-07-05T13:29:31.344Z [INFO] server-a32be3af-6d78-4033-c0b4-57ea88399745: added peer, starting replication: peer=server-3ab3d722-f0cf-bdbc-4734-416d6c849e90 -2024-07-05T13:29:31.344Z [INFO] server-3ab3d722-f0cf-bdbc-4734-416d6c849e90: duplicate requestVote for same term: term=2 -2024-07-05T13:29:31.344Z [INFO] server-a32be3af-6d78-4033-c0b4-57ea88399745: added peer, starting replication: peer=server-263bd370-a919-daf6-6105-bf3591640561 -2024-07-05T13:29:31.344Z [INFO] server-3ab3d722-f0cf-bdbc-4734-416d6c849e90: entering follower state: follower="Node at 3ab3d722-f0cf-bdbc-4734-416d6c849e90 [Follower]" leader-address=a32be3af-6d78-4033-c0b4-57ea88399745 leader-id=server-a32be3af-6d78-4033-c0b4-57ea88399745 -2024-07-05T13:29:31.344Z [INFO] server-a32be3af-6d78-4033-c0b4-57ea88399745: pipelining replication: peer="{Voter server-3ab3d722-f0cf-bdbc-4734-416d6c849e90 3ab3d722-f0cf-bdbc-4734-416d6c849e90}" -2024-07-05T13:29:31.344Z [INFO] server-a32be3af-6d78-4033-c0b4-57ea88399745: pipelining replication: peer="{Voter server-263bd370-a919-daf6-6105-bf3591640561 263bd370-a919-daf6-6105-bf3591640561}" -2024-07-05T13:29:31.344Z [WARN] server-263bd370-a919-daf6-6105-bf3591640561: rejecting vote request since we have a leader: from=3ab3d722-f0cf-bdbc-4734-416d6c849e90 leader=a32be3af-6d78-4033-c0b4-57ea88399745 leader-id=server-a32be3af-6d78-4033-c0b4-57ea88399745 -2024-07-05T13:29:31.450Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:31.344879093 +0000 UTC m=+12.114789768 (1 nodes), 70.613082ms from start of poll, 71.274739ms from cluster start. Timeout at 2024-07-05 13:29:31.449995166 +0000 UTC m=+12.219905841, 105.116073ms after stability -2024-07-05T13:29:31.450Z [INFO] server-a32be3af-6d78-4033-c0b4-57ea88399745: updating configuration: command=RemoveServer server-id=feafa643-80fb-70cb-8936-daba84920d06 server-addr= servers="[{Suffrage:Voter ID:server-3ab3d722-f0cf-bdbc-4734-416d6c849e90 Address:3ab3d722-f0cf-bdbc-4734-416d6c849e90} {Suffrage:Voter ID:server-263bd370-a919-daf6-6105-bf3591640561 Address:263bd370-a919-daf6-6105-bf3591640561} {Suffrage:Voter ID:server-a32be3af-6d78-4033-c0b4-57ea88399745 Address:a32be3af-6d78-4033-c0b4-57ea88399745}]" -2024-07-05T13:29:31.450Z [INFO] server-a32be3af-6d78-4033-c0b4-57ea88399745: aborting pipeline replication: peer="{Voter server-263bd370-a919-daf6-6105-bf3591640561 263bd370-a919-daf6-6105-bf3591640561}" -2024-07-05T13:29:31.450Z [INFO] server-a32be3af-6d78-4033-c0b4-57ea88399745: aborting pipeline replication: peer="{Voter server-3ab3d722-f0cf-bdbc-4734-416d6c849e90 3ab3d722-f0cf-bdbc-4734-416d6c849e90}" ---- PASS: TestRaft_RemoveUnknownPeer (0.18s) +2024-07-05T13:35:46.622Z [INFO] server-cfcb68fc-babb-02da-211c-45d6ed5e4b67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cfcb68fc-babb-02da-211c-45d6ed5e4b67 Address:cfcb68fc-babb-02da-211c-45d6ed5e4b67} {Suffrage:Voter ID:server-b8a44dff-9e44-9d4d-be5f-e5f999fb16c6 Address:b8a44dff-9e44-9d4d-be5f-e5f999fb16c6} {Suffrage:Voter ID:server-b71cee57-ef17-56ce-2931-5b45b5546e66 Address:b71cee57-ef17-56ce-2931-5b45b5546e66}]" +2024-07-05T13:35:46.622Z [INFO] server-cfcb68fc-babb-02da-211c-45d6ed5e4b67: entering follower state: follower="Node at cfcb68fc-babb-02da-211c-45d6ed5e4b67 [Follower]" leader-address= leader-id= +2024-07-05T13:35:46.623Z [INFO] server-b8a44dff-9e44-9d4d-be5f-e5f999fb16c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cfcb68fc-babb-02da-211c-45d6ed5e4b67 Address:cfcb68fc-babb-02da-211c-45d6ed5e4b67} {Suffrage:Voter ID:server-b8a44dff-9e44-9d4d-be5f-e5f999fb16c6 Address:b8a44dff-9e44-9d4d-be5f-e5f999fb16c6} {Suffrage:Voter ID:server-b71cee57-ef17-56ce-2931-5b45b5546e66 Address:b71cee57-ef17-56ce-2931-5b45b5546e66}]" +2024-07-05T13:35:46.623Z [INFO] server-b8a44dff-9e44-9d4d-be5f-e5f999fb16c6: entering follower state: follower="Node at b8a44dff-9e44-9d4d-be5f-e5f999fb16c6 [Follower]" leader-address= leader-id= +2024-07-05T13:35:46.624Z [INFO] server-b71cee57-ef17-56ce-2931-5b45b5546e66: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cfcb68fc-babb-02da-211c-45d6ed5e4b67 Address:cfcb68fc-babb-02da-211c-45d6ed5e4b67} {Suffrage:Voter ID:server-b8a44dff-9e44-9d4d-be5f-e5f999fb16c6 Address:b8a44dff-9e44-9d4d-be5f-e5f999fb16c6} {Suffrage:Voter ID:server-b71cee57-ef17-56ce-2931-5b45b5546e66 Address:b71cee57-ef17-56ce-2931-5b45b5546e66}]" +2024-07-05T13:35:46.624Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:46.624Z [INFO] server-b71cee57-ef17-56ce-2931-5b45b5546e66: entering follower state: follower="Node at b71cee57-ef17-56ce-2931-5b45b5546e66 [Follower]" leader-address= leader-id= +2024-07-05T13:35:46.694Z [WARN] server-cfcb68fc-babb-02da-211c-45d6ed5e4b67: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:46.694Z [INFO] server-cfcb68fc-babb-02da-211c-45d6ed5e4b67: entering candidate state: node="Node at cfcb68fc-babb-02da-211c-45d6ed5e4b67 [Candidate]" term=2 +2024-07-05T13:35:46.694Z [INFO] server-cfcb68fc-babb-02da-211c-45d6ed5e4b67: election won: term=2 tally=2 +2024-07-05T13:35:46.695Z [INFO] server-cfcb68fc-babb-02da-211c-45d6ed5e4b67: entering leader state: leader="Node at cfcb68fc-babb-02da-211c-45d6ed5e4b67 [Leader]" +2024-07-05T13:35:46.695Z [INFO] server-cfcb68fc-babb-02da-211c-45d6ed5e4b67: added peer, starting replication: peer=server-b8a44dff-9e44-9d4d-be5f-e5f999fb16c6 +2024-07-05T13:35:46.695Z [INFO] server-cfcb68fc-babb-02da-211c-45d6ed5e4b67: added peer, starting replication: peer=server-b71cee57-ef17-56ce-2931-5b45b5546e66 +2024-07-05T13:35:46.695Z [INFO] server-cfcb68fc-babb-02da-211c-45d6ed5e4b67: pipelining replication: peer="{Voter server-b8a44dff-9e44-9d4d-be5f-e5f999fb16c6 b8a44dff-9e44-9d4d-be5f-e5f999fb16c6}" +2024-07-05T13:35:46.695Z [INFO] server-cfcb68fc-babb-02da-211c-45d6ed5e4b67: pipelining replication: peer="{Voter server-b71cee57-ef17-56ce-2931-5b45b5546e66 b71cee57-ef17-56ce-2931-5b45b5546e66}" +2024-07-05T13:35:46.800Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:46.695024639 +0000 UTC m=+12.690945216 (1 nodes), 70.304966ms from start of poll, 73.233829ms from cluster start. Timeout at 2024-07-05 13:35:46.800058012 +0000 UTC m=+12.795979464, 105.034248ms after stability +2024-07-05T13:35:46.800Z [INFO] server-cfcb68fc-babb-02da-211c-45d6ed5e4b67: updating configuration: command=RemoveServer server-id=5f2e47ea-5802-a797-c6b4-a8d366d128e4 server-addr= servers="[{Suffrage:Voter ID:server-cfcb68fc-babb-02da-211c-45d6ed5e4b67 Address:cfcb68fc-babb-02da-211c-45d6ed5e4b67} {Suffrage:Voter ID:server-b8a44dff-9e44-9d4d-be5f-e5f999fb16c6 Address:b8a44dff-9e44-9d4d-be5f-e5f999fb16c6} {Suffrage:Voter ID:server-b71cee57-ef17-56ce-2931-5b45b5546e66 Address:b71cee57-ef17-56ce-2931-5b45b5546e66}]" +2024-07-05T13:35:46.802Z [INFO] server-cfcb68fc-babb-02da-211c-45d6ed5e4b67: aborting pipeline replication: peer="{Voter server-b71cee57-ef17-56ce-2931-5b45b5546e66 b71cee57-ef17-56ce-2931-5b45b5546e66}" +2024-07-05T13:35:46.802Z [INFO] server-cfcb68fc-babb-02da-211c-45d6ed5e4b67: aborting pipeline replication: peer="{Voter server-b8a44dff-9e44-9d4d-be5f-e5f999fb16c6 b8a44dff-9e44-9d4d-be5f-e5f999fb16c6}" +--- PASS: TestRaft_RemoveUnknownPeer (0.19s) === RUN TestRaft_SnapshotRestore -2024-07-05T13:29:31.452Z [INFO] server-4c7ff253-e431-ebbf-c957-a1ac364ed158: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c7ff253-e431-ebbf-c957-a1ac364ed158 Address:4c7ff253-e431-ebbf-c957-a1ac364ed158}]" -2024-07-05T13:29:31.452Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:31.452Z [INFO] server-4c7ff253-e431-ebbf-c957-a1ac364ed158: entering follower state: follower="Node at 4c7ff253-e431-ebbf-c957-a1ac364ed158 [Follower]" leader-address= leader-id= -2024-07-05T13:29:31.526Z [WARN] server-4c7ff253-e431-ebbf-c957-a1ac364ed158: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:31.526Z [INFO] server-4c7ff253-e431-ebbf-c957-a1ac364ed158: entering candidate state: node="Node at 4c7ff253-e431-ebbf-c957-a1ac364ed158 [Candidate]" term=2 -2024-07-05T13:29:31.526Z [INFO] server-4c7ff253-e431-ebbf-c957-a1ac364ed158: election won: term=2 tally=1 -2024-07-05T13:29:31.526Z [INFO] server-4c7ff253-e431-ebbf-c957-a1ac364ed158: entering leader state: leader="Node at 4c7ff253-e431-ebbf-c957-a1ac364ed158 [Leader]" -2024-07-05T13:29:31.632Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:31.526898428 +0000 UTC m=+12.296809103 (1 nodes), 74.306593ms from start of poll, 74.611143ms from cluster start. Timeout at 2024-07-05 13:29:31.632701436 +0000 UTC m=+12.402612111, 105.803008ms after stability -2024-07-05T13:29:31.640Z [INFO] server-4c7ff253-e431-ebbf-c957-a1ac364ed158: starting snapshot up to: index=102 -2024-07-05T13:29:31.640Z [INFO] creating new snapshot: path=/tmp/raft881532262/snapshots/2-102-1720186171640.tmp -2024-07-05T13:29:31.641Z [INFO] server-4c7ff253-e431-ebbf-c957-a1ac364ed158: compacting logs: from=1 to=92 -2024-07-05T13:29:31.641Z [INFO] server-4c7ff253-e431-ebbf-c957-a1ac364ed158: snapshot complete up to: index=102 -2024-07-05T13:29:31.642Z [INFO] server-4c7ff253-e431-ebbf-c957-a1ac364ed158: starting restore from snapshot: id=2-102-1720186171640 last-index=102 last-term=2 size-in-bytes=693 -2024-07-05T13:29:31.642Z [INFO] server-4c7ff253-e431-ebbf-c957-a1ac364ed158: snapshot restore progress: id=2-102-1720186171640 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% -2024-07-05T13:29:31.642Z [INFO] server-4c7ff253-e431-ebbf-c957-a1ac364ed158: restored from snapshot: id=2-102-1720186171640 last-index=102 last-term=2 size-in-bytes=693 -2024-07-05T13:29:31.642Z [INFO] server-4c7ff253-e431-ebbf-c957-a1ac364ed158: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c7ff253-e431-ebbf-c957-a1ac364ed158 Address:4c7ff253-e431-ebbf-c957-a1ac364ed158}]" ---- PASS: TestRaft_SnapshotRestore (0.19s) +2024-07-05T13:35:46.809Z [INFO] server-c8e37844-d4b0-a427-1ec1-4789302b2381: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8e37844-d4b0-a427-1ec1-4789302b2381 Address:c8e37844-d4b0-a427-1ec1-4789302b2381}]" +2024-07-05T13:35:46.810Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:46.810Z [INFO] server-c8e37844-d4b0-a427-1ec1-4789302b2381: entering follower state: follower="Node at c8e37844-d4b0-a427-1ec1-4789302b2381 [Follower]" leader-address= leader-id= +2024-07-05T13:35:46.875Z [WARN] server-c8e37844-d4b0-a427-1ec1-4789302b2381: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:46.876Z [INFO] server-c8e37844-d4b0-a427-1ec1-4789302b2381: entering candidate state: node="Node at c8e37844-d4b0-a427-1ec1-4789302b2381 [Candidate]" term=2 +2024-07-05T13:35:46.876Z [INFO] server-c8e37844-d4b0-a427-1ec1-4789302b2381: election won: term=2 tally=1 +2024-07-05T13:35:46.876Z [INFO] server-c8e37844-d4b0-a427-1ec1-4789302b2381: entering leader state: leader="Node at c8e37844-d4b0-a427-1ec1-4789302b2381 [Leader]" +2024-07-05T13:35:46.982Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:46.876581489 +0000 UTC m=+12.872502941 (1 nodes), 66.275664ms from start of poll, 67.393019ms from cluster start. Timeout at 2024-07-05 13:35:46.982248351 +0000 UTC m=+12.978169803, 105.666862ms after stability +2024-07-05T13:35:47.010Z [INFO] server-c8e37844-d4b0-a427-1ec1-4789302b2381: starting snapshot up to: index=102 +2024-07-05T13:35:47.010Z [INFO] creating new snapshot: path=/tmp/raft1258706142/snapshots/2-102-1720186547010.tmp +2024-07-05T13:35:47.014Z [INFO] server-c8e37844-d4b0-a427-1ec1-4789302b2381: compacting logs: from=1 to=92 +2024-07-05T13:35:47.014Z [INFO] server-c8e37844-d4b0-a427-1ec1-4789302b2381: snapshot complete up to: index=102 +2024-07-05T13:35:47.016Z [INFO] server-c8e37844-d4b0-a427-1ec1-4789302b2381: starting restore from snapshot: id=2-102-1720186547010 last-index=102 last-term=2 size-in-bytes=693 +2024-07-05T13:35:47.017Z [INFO] server-c8e37844-d4b0-a427-1ec1-4789302b2381: snapshot restore progress: id=2-102-1720186547010 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% +2024-07-05T13:35:47.018Z [INFO] server-c8e37844-d4b0-a427-1ec1-4789302b2381: restored from snapshot: id=2-102-1720186547010 last-index=102 last-term=2 size-in-bytes=693 +2024-07-05T13:35:47.018Z [INFO] server-c8e37844-d4b0-a427-1ec1-4789302b2381: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8e37844-d4b0-a427-1ec1-4789302b2381 Address:c8e37844-d4b0-a427-1ec1-4789302b2381}]" +--- PASS: TestRaft_SnapshotRestore (0.21s) === RUN TestRaft_RestoreSnapshotOnStartup_Monotonic -2024-07-05T13:29:31.646Z [INFO] server-a9bfb6fa-3881-ef79-8d04-51233a975a40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9bfb6fa-3881-ef79-8d04-51233a975a40 Address:a9bfb6fa-3881-ef79-8d04-51233a975a40}]" -2024-07-05T13:29:31.646Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:31.646Z [INFO] server-a9bfb6fa-3881-ef79-8d04-51233a975a40: entering follower state: follower="Node at a9bfb6fa-3881-ef79-8d04-51233a975a40 [Follower]" leader-address= leader-id= -2024-07-05T13:29:31.735Z [WARN] server-a9bfb6fa-3881-ef79-8d04-51233a975a40: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:31.735Z [INFO] server-a9bfb6fa-3881-ef79-8d04-51233a975a40: entering candidate state: node="Node at a9bfb6fa-3881-ef79-8d04-51233a975a40 [Candidate]" term=2 -2024-07-05T13:29:31.735Z [INFO] server-a9bfb6fa-3881-ef79-8d04-51233a975a40: election won: term=2 tally=1 -2024-07-05T13:29:31.735Z [INFO] server-a9bfb6fa-3881-ef79-8d04-51233a975a40: entering leader state: leader="Node at a9bfb6fa-3881-ef79-8d04-51233a975a40 [Leader]" -2024-07-05T13:29:31.841Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:31.735305317 +0000 UTC m=+12.505215992 (1 nodes), 89.181791ms from start of poll, 89.442182ms from cluster start. Timeout at 2024-07-05 13:29:31.841389875 +0000 UTC m=+12.611300550, 106.084558ms after stability -2024-07-05T13:29:31.846Z [INFO] server-a9bfb6fa-3881-ef79-8d04-51233a975a40: starting snapshot up to: index=102 -2024-07-05T13:29:31.846Z [INFO] creating new snapshot: path=/tmp/raft3104833864/snapshots/2-102-1720186171846.tmp -2024-07-05T13:29:31.847Z [INFO] server-a9bfb6fa-3881-ef79-8d04-51233a975a40: compacting logs: from=1 to=92 -2024-07-05T13:29:31.847Z [INFO] server-a9bfb6fa-3881-ef79-8d04-51233a975a40: snapshot complete up to: index=102 -2024-07-05T13:29:31.848Z [INFO] server-a9bfb6fa-3881-ef79-8d04-51233a975a40: starting restore from snapshot: id=2-102-1720186171846 last-index=102 last-term=2 size-in-bytes=693 -2024-07-05T13:29:31.848Z [INFO] server-a9bfb6fa-3881-ef79-8d04-51233a975a40: snapshot restore progress: id=2-102-1720186171846 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% -2024-07-05T13:29:31.848Z [INFO] server-a9bfb6fa-3881-ef79-8d04-51233a975a40: restored from snapshot: id=2-102-1720186171846 last-index=102 last-term=2 size-in-bytes=693 -2024-07-05T13:29:31.848Z [INFO] server-a9bfb6fa-3881-ef79-8d04-51233a975a40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9bfb6fa-3881-ef79-8d04-51233a975a40 Address:a9bfb6fa-3881-ef79-8d04-51233a975a40}]" ---- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.20s) +2024-07-05T13:35:47.023Z [INFO] server-0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6 Address:0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6}]" +2024-07-05T13:35:47.024Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:47.024Z [INFO] server-0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6: entering follower state: follower="Node at 0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6 [Follower]" leader-address= leader-id= +2024-07-05T13:35:47.093Z [WARN] server-0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:47.094Z [INFO] server-0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6: entering candidate state: node="Node at 0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6 [Candidate]" term=2 +2024-07-05T13:35:47.094Z [INFO] server-0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6: election won: term=2 tally=1 +2024-07-05T13:35:47.094Z [INFO] server-0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6: entering leader state: leader="Node at 0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6 [Leader]" +2024-07-05T13:35:47.200Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:47.094589882 +0000 UTC m=+13.090511626 (1 nodes), 70.47938ms from start of poll, 71.372155ms from cluster start. Timeout at 2024-07-05 13:35:47.200248286 +0000 UTC m=+13.196169737, 105.658111ms after stability +2024-07-05T13:35:47.229Z [INFO] server-0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6: starting snapshot up to: index=102 +2024-07-05T13:35:47.229Z [INFO] creating new snapshot: path=/tmp/raft1586531325/snapshots/2-102-1720186547229.tmp +2024-07-05T13:35:47.233Z [INFO] server-0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6: compacting logs: from=1 to=92 +2024-07-05T13:35:47.233Z [INFO] server-0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6: snapshot complete up to: index=102 +2024-07-05T13:35:47.235Z [INFO] server-0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6: starting restore from snapshot: id=2-102-1720186547229 last-index=102 last-term=2 size-in-bytes=693 +2024-07-05T13:35:47.238Z [INFO] server-0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6: snapshot restore progress: id=2-102-1720186547229 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% +2024-07-05T13:35:47.239Z [INFO] server-0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6: restored from snapshot: id=2-102-1720186547229 last-index=102 last-term=2 size-in-bytes=693 +2024-07-05T13:35:47.239Z [INFO] server-0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6 Address:0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6}]" +2024-07-05T13:35:47.239Z [INFO] server-0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6: entering follower state: follower="Node at 0dab1264-e0e7-0bf0-1b46-b4807ad9dfc6 [Shutdown]" leader-address= leader-id= +--- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.22s) === RUN TestRaft_SnapshotRestore_Progress -2024-07-05T13:29:31.850Z [INFO] server-fac2ba7a-1a24-4c7a-b2c4-b10897181d19: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fac2ba7a-1a24-4c7a-b2c4-b10897181d19 Address:fac2ba7a-1a24-4c7a-b2c4-b10897181d19}]" -2024-07-05T13:29:31.850Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:31.850Z [INFO] server-fac2ba7a-1a24-4c7a-b2c4-b10897181d19: entering follower state: follower="Node at fac2ba7a-1a24-4c7a-b2c4-b10897181d19 [Follower]" leader-address= leader-id= -2024-07-05T13:29:31.912Z [WARN] server-fac2ba7a-1a24-4c7a-b2c4-b10897181d19: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:31.912Z [INFO] server-fac2ba7a-1a24-4c7a-b2c4-b10897181d19: entering candidate state: node="Node at fac2ba7a-1a24-4c7a-b2c4-b10897181d19 [Candidate]" term=2 -2024-07-05T13:29:31.913Z [INFO] server-fac2ba7a-1a24-4c7a-b2c4-b10897181d19: election won: term=2 tally=1 -2024-07-05T13:29:31.913Z [INFO] server-fac2ba7a-1a24-4c7a-b2c4-b10897181d19: entering leader state: leader="Node at fac2ba7a-1a24-4c7a-b2c4-b10897181d19 [Leader]" -2024-07-05T13:29:32.018Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:31.913171757 +0000 UTC m=+12.683082432 (1 nodes), 63.016149ms from start of poll, 63.231542ms from cluster start. Timeout at 2024-07-05 13:29:32.01826863 +0000 UTC m=+12.788179305, 105.096873ms after stability -2024-07-05T13:29:32.023Z [INFO] server-fac2ba7a-1a24-4c7a-b2c4-b10897181d19: starting snapshot up to: index=102 -2024-07-05T13:29:32.023Z [INFO] creating new snapshot: path=/tmp/raft1595026449/snapshots/2-102-1720186172023.tmp -2024-07-05T13:29:32.024Z [INFO] server-fac2ba7a-1a24-4c7a-b2c4-b10897181d19: compacting logs: from=1 to=92 -2024-07-05T13:29:32.024Z [INFO] server-fac2ba7a-1a24-4c7a-b2c4-b10897181d19: snapshot complete up to: index=102 ---- PASS: TestRaft_SnapshotRestore_Progress (0.18s) +2024-07-05T13:35:47.247Z [INFO] server-6ad23d09-24c3-d22b-d5fb-bc306959fe6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ad23d09-24c3-d22b-d5fb-bc306959fe6f Address:6ad23d09-24c3-d22b-d5fb-bc306959fe6f}]" +2024-07-05T13:35:47.247Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:47.247Z [INFO] server-6ad23d09-24c3-d22b-d5fb-bc306959fe6f: entering follower state: follower="Node at 6ad23d09-24c3-d22b-d5fb-bc306959fe6f [Follower]" leader-address= leader-id= +2024-07-05T13:35:47.305Z [WARN] server-6ad23d09-24c3-d22b-d5fb-bc306959fe6f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:47.306Z [INFO] server-6ad23d09-24c3-d22b-d5fb-bc306959fe6f: entering candidate state: node="Node at 6ad23d09-24c3-d22b-d5fb-bc306959fe6f [Candidate]" term=2 +2024-07-05T13:35:47.306Z [INFO] server-6ad23d09-24c3-d22b-d5fb-bc306959fe6f: election won: term=2 tally=1 +2024-07-05T13:35:47.306Z [INFO] server-6ad23d09-24c3-d22b-d5fb-bc306959fe6f: entering leader state: leader="Node at 6ad23d09-24c3-d22b-d5fb-bc306959fe6f [Leader]" +2024-07-05T13:35:47.412Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:47.306789256 +0000 UTC m=+13.302710999 (1 nodes), 59.349874ms from start of poll, 60.313231ms from cluster start. Timeout at 2024-07-05 13:35:47.412613323 +0000 UTC m=+13.408534775, 105.823776ms after stability +2024-07-05T13:35:47.441Z [INFO] server-6ad23d09-24c3-d22b-d5fb-bc306959fe6f: starting snapshot up to: index=102 +2024-07-05T13:35:47.441Z [INFO] creating new snapshot: path=/tmp/raft3470268363/snapshots/2-102-1720186547441.tmp +2024-07-05T13:35:47.445Z [INFO] server-6ad23d09-24c3-d22b-d5fb-bc306959fe6f: compacting logs: from=1 to=92 +2024-07-05T13:35:47.445Z [INFO] server-6ad23d09-24c3-d22b-d5fb-bc306959fe6f: snapshot complete up to: index=102 +--- PASS: TestRaft_SnapshotRestore_Progress (0.21s) === RUN TestRaft_NoRestoreOnStart -2024-07-05T13:29:32.026Z [INFO] server-3ea3f666-0284-cd38-0308-c3f89ce2ee84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ea3f666-0284-cd38-0308-c3f89ce2ee84 Address:3ea3f666-0284-cd38-0308-c3f89ce2ee84}]" -2024-07-05T13:29:32.026Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:32.026Z [INFO] server-3ea3f666-0284-cd38-0308-c3f89ce2ee84: entering follower state: follower="Node at 3ea3f666-0284-cd38-0308-c3f89ce2ee84 [Follower]" leader-address= leader-id= -2024-07-05T13:29:32.117Z [WARN] server-3ea3f666-0284-cd38-0308-c3f89ce2ee84: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:32.117Z [INFO] server-3ea3f666-0284-cd38-0308-c3f89ce2ee84: entering candidate state: node="Node at 3ea3f666-0284-cd38-0308-c3f89ce2ee84 [Candidate]" term=2 -2024-07-05T13:29:32.117Z [INFO] server-3ea3f666-0284-cd38-0308-c3f89ce2ee84: election won: term=2 tally=1 -2024-07-05T13:29:32.117Z [INFO] server-3ea3f666-0284-cd38-0308-c3f89ce2ee84: entering leader state: leader="Node at 3ea3f666-0284-cd38-0308-c3f89ce2ee84 [Leader]" -2024-07-05T13:29:32.223Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:32.11804309 +0000 UTC m=+12.887953765 (1 nodes), 91.238119ms from start of poll, 91.461591ms from cluster start. Timeout at 2024-07-05 13:29:32.223731023 +0000 UTC m=+12.993641778, 105.688013ms after stability -2024-07-05T13:29:32.229Z [INFO] server-3ea3f666-0284-cd38-0308-c3f89ce2ee84: starting snapshot up to: index=102 -2024-07-05T13:29:32.229Z [INFO] creating new snapshot: path=/tmp/raft3733366984/snapshots/2-102-1720186172229.tmp -2024-07-05T13:29:32.230Z [INFO] server-3ea3f666-0284-cd38-0308-c3f89ce2ee84: compacting logs: from=1 to=92 -2024-07-05T13:29:32.230Z [INFO] server-3ea3f666-0284-cd38-0308-c3f89ce2ee84: snapshot complete up to: index=102 -2024-07-05T13:29:32.231Z [INFO] server-3ea3f666-0284-cd38-0308-c3f89ce2ee84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ea3f666-0284-cd38-0308-c3f89ce2ee84 Address:3ea3f666-0284-cd38-0308-c3f89ce2ee84}]" ---- PASS: TestRaft_NoRestoreOnStart (0.20s) +2024-07-05T13:35:47.455Z [INFO] server-5d97b202-7f41-75ce-04ac-c7353ff90fed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5d97b202-7f41-75ce-04ac-c7353ff90fed Address:5d97b202-7f41-75ce-04ac-c7353ff90fed}]" +2024-07-05T13:35:47.455Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:47.455Z [INFO] server-5d97b202-7f41-75ce-04ac-c7353ff90fed: entering follower state: follower="Node at 5d97b202-7f41-75ce-04ac-c7353ff90fed [Follower]" leader-address= leader-id= +2024-07-05T13:35:47.556Z [WARN] server-5d97b202-7f41-75ce-04ac-c7353ff90fed: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:47.556Z [INFO] server-5d97b202-7f41-75ce-04ac-c7353ff90fed: entering candidate state: node="Node at 5d97b202-7f41-75ce-04ac-c7353ff90fed [Candidate]" term=2 +2024-07-05T13:35:47.556Z [INFO] server-5d97b202-7f41-75ce-04ac-c7353ff90fed: election won: term=2 tally=1 +2024-07-05T13:35:47.556Z [INFO] server-5d97b202-7f41-75ce-04ac-c7353ff90fed: entering leader state: leader="Node at 5d97b202-7f41-75ce-04ac-c7353ff90fed [Leader]" +2024-07-05T13:35:47.662Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:47.557103517 +0000 UTC m=+13.553024969 (1 nodes), 101.180527ms from start of poll, 102.114718ms from cluster start. Timeout at 2024-07-05 13:35:47.662147973 +0000 UTC m=+13.658069425, 105.044456ms after stability +2024-07-05T13:35:47.689Z [INFO] server-5d97b202-7f41-75ce-04ac-c7353ff90fed: starting snapshot up to: index=102 +2024-07-05T13:35:47.689Z [INFO] creating new snapshot: path=/tmp/raft4249336708/snapshots/2-102-1720186547689.tmp +2024-07-05T13:35:47.693Z [INFO] server-5d97b202-7f41-75ce-04ac-c7353ff90fed: compacting logs: from=1 to=92 +2024-07-05T13:35:47.693Z [INFO] server-5d97b202-7f41-75ce-04ac-c7353ff90fed: snapshot complete up to: index=102 +2024-07-05T13:35:47.694Z [INFO] server-5d97b202-7f41-75ce-04ac-c7353ff90fed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5d97b202-7f41-75ce-04ac-c7353ff90fed Address:5d97b202-7f41-75ce-04ac-c7353ff90fed}]" +--- PASS: TestRaft_NoRestoreOnStart (0.24s) === RUN TestRaft_SnapshotRestore_PeerChange raft_test.go:1266: Skipping unreliable test +2024-07-05T13:35:47.695Z [INFO] server-5d97b202-7f41-75ce-04ac-c7353ff90fed: entering follower state: follower="Node at 5d97b202-7f41-75ce-04ac-c7353ff90fed [Follower]" leader-address= leader-id= --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot -2024-07-05T13:29:32.231Z [INFO] server-371d614e-683d-467e-013c-b576855b1d13: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-371d614e-683d-467e-013c-b576855b1d13 Address:371d614e-683d-467e-013c-b576855b1d13}]" -2024-07-05T13:29:32.231Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:32.232Z [INFO] server-3ea3f666-0284-cd38-0308-c3f89ce2ee84: entering follower state: follower="Node at 3ea3f666-0284-cd38-0308-c3f89ce2ee84 [Follower]" leader-address= leader-id= -2024-07-05T13:29:32.232Z [INFO] server-371d614e-683d-467e-013c-b576855b1d13: entering follower state: follower="Node at 371d614e-683d-467e-013c-b576855b1d13 [Follower]" leader-address= leader-id= -2024-07-05T13:29:32.288Z [WARN] server-371d614e-683d-467e-013c-b576855b1d13: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:32.288Z [INFO] server-371d614e-683d-467e-013c-b576855b1d13: entering candidate state: node="Node at 371d614e-683d-467e-013c-b576855b1d13 [Candidate]" term=2 -2024-07-05T13:29:32.288Z [INFO] server-371d614e-683d-467e-013c-b576855b1d13: election won: term=2 tally=1 -2024-07-05T13:29:32.288Z [INFO] server-371d614e-683d-467e-013c-b576855b1d13: entering leader state: leader="Node at 371d614e-683d-467e-013c-b576855b1d13 [Leader]" -2024-07-05T13:29:32.296Z [WARN] server-3ea3f666-0284-cd38-0308-c3f89ce2ee84: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:32.296Z [INFO] server-3ea3f666-0284-cd38-0308-c3f89ce2ee84: entering candidate state: node="Node at 3ea3f666-0284-cd38-0308-c3f89ce2ee84 [Candidate]" term=3 -2024-07-05T13:29:32.296Z [INFO] server-3ea3f666-0284-cd38-0308-c3f89ce2ee84: election won: term=3 tally=1 -2024-07-05T13:29:32.296Z [INFO] server-3ea3f666-0284-cd38-0308-c3f89ce2ee84: entering leader state: leader="Node at 3ea3f666-0284-cd38-0308-c3f89ce2ee84 [Leader]" -2024-07-05T13:29:32.396Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:32.288616506 +0000 UTC m=+13.058527181 (1 nodes), 56.604894ms from start of poll, 56.895804ms from cluster start. Timeout at 2024-07-05 13:29:32.396199972 +0000 UTC m=+13.166110687, 107.583506ms after stability -2024-07-05T13:29:32.401Z [INFO] server-371d614e-683d-467e-013c-b576855b1d13: starting snapshot up to: index=90 -2024-07-05T13:29:32.401Z [INFO] creating new snapshot: path=/tmp/raft3362649711/snapshots/2-90-1720186172401.tmp -2024-07-05T13:29:32.402Z [INFO] server-371d614e-683d-467e-013c-b576855b1d13: compacting logs: from=1 to=90 -2024-07-05T13:29:32.402Z [INFO] server-371d614e-683d-467e-013c-b576855b1d13: snapshot complete up to: index=90 ---- PASS: TestRaft_AutoSnapshot (0.28s) +2024-07-05T13:35:47.695Z [INFO] server-96596794-8a54-5b0a-7a91-f0d9b21ef2c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96596794-8a54-5b0a-7a91-f0d9b21ef2c6 Address:96596794-8a54-5b0a-7a91-f0d9b21ef2c6}]" +2024-07-05T13:35:47.695Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:47.696Z [INFO] server-96596794-8a54-5b0a-7a91-f0d9b21ef2c6: entering follower state: follower="Node at 96596794-8a54-5b0a-7a91-f0d9b21ef2c6 [Follower]" leader-address= leader-id= +2024-07-05T13:35:47.755Z [WARN] server-96596794-8a54-5b0a-7a91-f0d9b21ef2c6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:47.756Z [INFO] server-96596794-8a54-5b0a-7a91-f0d9b21ef2c6: entering candidate state: node="Node at 96596794-8a54-5b0a-7a91-f0d9b21ef2c6 [Candidate]" term=2 +2024-07-05T13:35:47.756Z [INFO] server-96596794-8a54-5b0a-7a91-f0d9b21ef2c6: election won: term=2 tally=1 +2024-07-05T13:35:47.756Z [INFO] server-96596794-8a54-5b0a-7a91-f0d9b21ef2c6: entering leader state: leader="Node at 96596794-8a54-5b0a-7a91-f0d9b21ef2c6 [Leader]" +2024-07-05T13:35:47.774Z [WARN] server-5d97b202-7f41-75ce-04ac-c7353ff90fed: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:47.774Z [INFO] server-5d97b202-7f41-75ce-04ac-c7353ff90fed: entering candidate state: node="Node at 5d97b202-7f41-75ce-04ac-c7353ff90fed [Candidate]" term=3 +2024-07-05T13:35:47.775Z [INFO] server-5d97b202-7f41-75ce-04ac-c7353ff90fed: election won: term=3 tally=1 +2024-07-05T13:35:47.775Z [INFO] server-5d97b202-7f41-75ce-04ac-c7353ff90fed: entering leader state: leader="Node at 5d97b202-7f41-75ce-04ac-c7353ff90fed [Leader]" +2024-07-05T13:35:47.862Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:47.75676787 +0000 UTC m=+13.752689613 (1 nodes), 60.768515ms from start of poll, 61.063093ms from cluster start. Timeout at 2024-07-05 13:35:47.862367066 +0000 UTC m=+13.858288517, 105.598904ms after stability +2024-07-05T13:35:47.887Z [INFO] server-96596794-8a54-5b0a-7a91-f0d9b21ef2c6: starting snapshot up to: index=86 +2024-07-05T13:35:47.887Z [INFO] creating new snapshot: path=/tmp/raft3333033020/snapshots/2-86-1720186547887.tmp +2024-07-05T13:35:47.891Z [INFO] server-96596794-8a54-5b0a-7a91-f0d9b21ef2c6: compacting logs: from=1 to=86 +2024-07-05T13:35:47.891Z [INFO] server-96596794-8a54-5b0a-7a91-f0d9b21ef2c6: snapshot complete up to: index=86 +--- PASS: TestRaft_AutoSnapshot (0.31s) === RUN TestRaft_UserSnapshot -2024-07-05T13:29:32.508Z [INFO] server-36630d72-8fe9-1394-c9b6-723fb83a2cbe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36630d72-8fe9-1394-c9b6-723fb83a2cbe Address:36630d72-8fe9-1394-c9b6-723fb83a2cbe}]" -2024-07-05T13:29:32.508Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:32.508Z [INFO] server-36630d72-8fe9-1394-c9b6-723fb83a2cbe: entering follower state: follower="Node at 36630d72-8fe9-1394-c9b6-723fb83a2cbe [Follower]" leader-address= leader-id= -2024-07-05T13:29:32.584Z [WARN] server-36630d72-8fe9-1394-c9b6-723fb83a2cbe: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:32.584Z [INFO] server-36630d72-8fe9-1394-c9b6-723fb83a2cbe: entering candidate state: node="Node at 36630d72-8fe9-1394-c9b6-723fb83a2cbe [Candidate]" term=2 -2024-07-05T13:29:32.584Z [INFO] server-36630d72-8fe9-1394-c9b6-723fb83a2cbe: election won: term=2 tally=1 -2024-07-05T13:29:32.584Z [INFO] server-36630d72-8fe9-1394-c9b6-723fb83a2cbe: entering leader state: leader="Node at 36630d72-8fe9-1394-c9b6-723fb83a2cbe [Leader]" -2024-07-05T13:29:32.692Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:32.584787356 +0000 UTC m=+13.354698031 (1 nodes), 76.247965ms from start of poll, 76.555354ms from cluster start. Timeout at 2024-07-05 13:29:32.692181749 +0000 UTC m=+13.462092464, 107.394433ms after stability -2024-07-05T13:29:32.692Z [ERROR] server-36630d72-8fe9-1394-c9b6-723fb83a2cbe: failed to take snapshot: error="nothing new to snapshot" -2024-07-05T13:29:32.693Z [INFO] server-36630d72-8fe9-1394-c9b6-723fb83a2cbe: starting snapshot up to: index=12 -2024-07-05T13:29:32.693Z [INFO] creating new snapshot: path=/tmp/raft1157886857/snapshots/2-12-1720186172693.tmp -2024-07-05T13:29:32.694Z [INFO] server-36630d72-8fe9-1394-c9b6-723fb83a2cbe: compacting logs: from=1 to=2 -2024-07-05T13:29:32.694Z [INFO] server-36630d72-8fe9-1394-c9b6-723fb83a2cbe: snapshot complete up to: index=12 ---- PASS: TestRaft_UserSnapshot (0.19s) +2024-07-05T13:35:48.003Z [INFO] server-aab3ea98-1c0a-6a5f-b6c3-a61658f2a87d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aab3ea98-1c0a-6a5f-b6c3-a61658f2a87d Address:aab3ea98-1c0a-6a5f-b6c3-a61658f2a87d}]" +2024-07-05T13:35:48.004Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:48.004Z [INFO] server-aab3ea98-1c0a-6a5f-b6c3-a61658f2a87d: entering follower state: follower="Node at aab3ea98-1c0a-6a5f-b6c3-a61658f2a87d [Follower]" leader-address= leader-id= +2024-07-05T13:35:48.094Z [WARN] server-aab3ea98-1c0a-6a5f-b6c3-a61658f2a87d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:48.094Z [INFO] server-aab3ea98-1c0a-6a5f-b6c3-a61658f2a87d: entering candidate state: node="Node at aab3ea98-1c0a-6a5f-b6c3-a61658f2a87d [Candidate]" term=2 +2024-07-05T13:35:48.094Z [INFO] server-aab3ea98-1c0a-6a5f-b6c3-a61658f2a87d: election won: term=2 tally=1 +2024-07-05T13:35:48.094Z [INFO] server-aab3ea98-1c0a-6a5f-b6c3-a61658f2a87d: entering leader state: leader="Node at aab3ea98-1c0a-6a5f-b6c3-a61658f2a87d [Leader]" +2024-07-05T13:35:48.201Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:48.094832236 +0000 UTC m=+14.090753688 (1 nodes), 90.548888ms from start of poll, 91.524495ms from cluster start. Timeout at 2024-07-05 13:35:48.200937465 +0000 UTC m=+14.196858916, 106.105228ms after stability +2024-07-05T13:35:48.201Z [ERROR] server-aab3ea98-1c0a-6a5f-b6c3-a61658f2a87d: failed to take snapshot: error="nothing new to snapshot" +2024-07-05T13:35:48.205Z [INFO] server-aab3ea98-1c0a-6a5f-b6c3-a61658f2a87d: starting snapshot up to: index=12 +2024-07-05T13:35:48.205Z [INFO] creating new snapshot: path=/tmp/raft2456053882/snapshots/2-12-1720186548205.tmp +2024-07-05T13:35:48.208Z [INFO] server-aab3ea98-1c0a-6a5f-b6c3-a61658f2a87d: compacting logs: from=1 to=2 +2024-07-05T13:35:48.208Z [INFO] server-aab3ea98-1c0a-6a5f-b6c3-a61658f2a87d: snapshot complete up to: index=12 +--- PASS: TestRaft_UserSnapshot (0.21s) === RUN TestRaft_UserRestore === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false -2024-07-05T13:29:32.696Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f28d662-5321-68b5-15f7-7ea41ae93783 Address:5f28d662-5321-68b5-15f7-7ea41ae93783} {Suffrage:Voter ID:server-8de2ccf6-56cc-a69d-f958-907c8da8cd40 Address:8de2ccf6-56cc-a69d-f958-907c8da8cd40} {Suffrage:Voter ID:server-62a81957-35bf-f986-59fd-6a9cfcce2f1b Address:62a81957-35bf-f986-59fd-6a9cfcce2f1b}]" -2024-07-05T13:29:32.696Z [INFO] server-8de2ccf6-56cc-a69d-f958-907c8da8cd40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f28d662-5321-68b5-15f7-7ea41ae93783 Address:5f28d662-5321-68b5-15f7-7ea41ae93783} {Suffrage:Voter ID:server-8de2ccf6-56cc-a69d-f958-907c8da8cd40 Address:8de2ccf6-56cc-a69d-f958-907c8da8cd40} {Suffrage:Voter ID:server-62a81957-35bf-f986-59fd-6a9cfcce2f1b Address:62a81957-35bf-f986-59fd-6a9cfcce2f1b}]" -2024-07-05T13:29:32.696Z [INFO] server-62a81957-35bf-f986-59fd-6a9cfcce2f1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f28d662-5321-68b5-15f7-7ea41ae93783 Address:5f28d662-5321-68b5-15f7-7ea41ae93783} {Suffrage:Voter ID:server-8de2ccf6-56cc-a69d-f958-907c8da8cd40 Address:8de2ccf6-56cc-a69d-f958-907c8da8cd40} {Suffrage:Voter ID:server-62a81957-35bf-f986-59fd-6a9cfcce2f1b Address:62a81957-35bf-f986-59fd-6a9cfcce2f1b}]" -2024-07-05T13:29:32.696Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:32.697Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: entering follower state: follower="Node at 5f28d662-5321-68b5-15f7-7ea41ae93783 [Follower]" leader-address= leader-id= -2024-07-05T13:29:32.697Z [INFO] server-8de2ccf6-56cc-a69d-f958-907c8da8cd40: entering follower state: follower="Node at 8de2ccf6-56cc-a69d-f958-907c8da8cd40 [Follower]" leader-address= leader-id= -2024-07-05T13:29:32.697Z [INFO] server-62a81957-35bf-f986-59fd-6a9cfcce2f1b: entering follower state: follower="Node at 62a81957-35bf-f986-59fd-6a9cfcce2f1b [Follower]" leader-address= leader-id= -2024-07-05T13:29:33.301Z [WARN] server-5f28d662-5321-68b5-15f7-7ea41ae93783: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:33.301Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: entering candidate state: node="Node at 5f28d662-5321-68b5-15f7-7ea41ae93783 [Candidate]" term=2 -2024-07-05T13:29:33.301Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: election won: term=2 tally=2 -2024-07-05T13:29:33.301Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: entering leader state: leader="Node at 5f28d662-5321-68b5-15f7-7ea41ae93783 [Leader]" -2024-07-05T13:29:33.301Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: added peer, starting replication: peer=server-8de2ccf6-56cc-a69d-f958-907c8da8cd40 -2024-07-05T13:29:33.301Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: added peer, starting replication: peer=server-62a81957-35bf-f986-59fd-6a9cfcce2f1b -2024-07-05T13:29:33.301Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: pipelining replication: peer="{Voter server-62a81957-35bf-f986-59fd-6a9cfcce2f1b 62a81957-35bf-f986-59fd-6a9cfcce2f1b}" -2024-07-05T13:29:33.302Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: pipelining replication: peer="{Voter server-8de2ccf6-56cc-a69d-f958-907c8da8cd40 8de2ccf6-56cc-a69d-f958-907c8da8cd40}" -2024-07-05T13:29:34.308Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:33.302330025 +0000 UTC m=+14.072240700 (1 nodes), 605.324685ms from start of poll, 606.398008ms from cluster start. Timeout at 2024-07-05 13:29:34.308287615 +0000 UTC m=+15.078198250, 1.00595755s after stability -2024-07-05T13:29:34.309Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: starting snapshot up to: index=12 -2024-07-05T13:29:34.309Z [INFO] creating new snapshot: path=/tmp/raft998817544/snapshots/2-12-1720186174309.tmp -2024-07-05T13:29:34.311Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: snapshot complete up to: index=12 -2024-07-05T13:29:34.313Z [INFO] creating new snapshot: path=/tmp/raft998817544/snapshots/2-23-1720186174313.tmp -2024-07-05T13:29:34.314Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: copied to local snapshot: bytes=71 -2024-07-05T13:29:34.314Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: snapshot restore progress: id=2-23-1720186174313 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:34.314Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: restored user snapshot: index=23 -2024-07-05T13:29:34.314Z [ERROR] server-5f28d662-5321-68b5-15f7-7ea41ae93783: failed to get log: index=23 error="log not found" -2024-07-05T13:29:34.314Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: aborting pipeline replication: peer="{Voter server-62a81957-35bf-f986-59fd-6a9cfcce2f1b 62a81957-35bf-f986-59fd-6a9cfcce2f1b}" -2024-07-05T13:29:34.315Z [ERROR] server-5f28d662-5321-68b5-15f7-7ea41ae93783: failed to get log: index=23 error="log not found" -2024-07-05T13:29:34.315Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: aborting pipeline replication: peer="{Voter server-8de2ccf6-56cc-a69d-f958-907c8da8cd40 8de2ccf6-56cc-a69d-f958-907c8da8cd40}" -2024-07-05T13:29:34.321Z [ERROR] server-5f28d662-5321-68b5-15f7-7ea41ae93783: failed to get log: index=23 error="log not found" -2024-07-05T13:29:34.322Z [INFO] creating new snapshot: path=/tmp/raft3540611606/snapshots/2-23-1720186174322.tmp -2024-07-05T13:29:34.323Z [INFO] server-8de2ccf6-56cc-a69d-f958-907c8da8cd40: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:34.323Z [INFO] server-8de2ccf6-56cc-a69d-f958-907c8da8cd40: copied to local snapshot: bytes=71 -2024-07-05T13:29:34.324Z [ERROR] server-5f28d662-5321-68b5-15f7-7ea41ae93783: failed to get log: index=23 error="log not found" -2024-07-05T13:29:34.325Z [INFO] creating new snapshot: path=/tmp/raft851883072/snapshots/2-23-1720186174325.tmp -2024-07-05T13:29:34.325Z [INFO] server-62a81957-35bf-f986-59fd-6a9cfcce2f1b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:34.326Z [INFO] server-62a81957-35bf-f986-59fd-6a9cfcce2f1b: copied to local snapshot: bytes=71 -2024-07-05T13:29:34.326Z [INFO] server-62a81957-35bf-f986-59fd-6a9cfcce2f1b: snapshot restore progress: id=2-23-1720186174325 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:34.326Z [INFO] server-62a81957-35bf-f986-59fd-6a9cfcce2f1b: Installed remote snapshot -2024-07-05T13:29:34.327Z [WARN] server-62a81957-35bf-f986-59fd-6a9cfcce2f1b: skipping application of old log: index=22 -2024-07-05T13:29:34.327Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: pipelining replication: peer="{Voter server-62a81957-35bf-f986-59fd-6a9cfcce2f1b 62a81957-35bf-f986-59fd-6a9cfcce2f1b}" -2024-07-05T13:29:34.327Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:29:34.327Z [INFO] server-8de2ccf6-56cc-a69d-f958-907c8da8cd40: snapshot restore progress: id=2-23-1720186174322 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:34.327Z [INFO] server-8de2ccf6-56cc-a69d-f958-907c8da8cd40: Installed remote snapshot -2024-07-05T13:29:34.328Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: pipelining replication: peer="{Voter server-8de2ccf6-56cc-a69d-f958-907c8da8cd40 8de2ccf6-56cc-a69d-f958-907c8da8cd40}" -2024-07-05T13:29:35.333Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:29:34.327315907 +0000 UTC m=+15.097226582 (2 nodes), 2.04µs from start of poll, 1.63138389s from cluster start. Timeout at 2024-07-05 13:29:35.333132462 +0000 UTC m=+16.103043137, 1.005816555s after stability -2024-07-05T13:29:35.346Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: aborting pipeline replication: peer="{Voter server-62a81957-35bf-f986-59fd-6a9cfcce2f1b 62a81957-35bf-f986-59fd-6a9cfcce2f1b}" -2024-07-05T13:29:35.346Z [INFO] server-5f28d662-5321-68b5-15f7-7ea41ae93783: aborting pipeline replication: peer="{Voter server-8de2ccf6-56cc-a69d-f958-907c8da8cd40 8de2ccf6-56cc-a69d-f958-907c8da8cd40}" +2024-07-05T13:35:48.218Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ab67aa8e-194d-20d7-4458-cd7d8969b438 Address:ab67aa8e-194d-20d7-4458-cd7d8969b438} {Suffrage:Voter ID:server-b5d7480d-060b-9998-56c6-b9848693bb43 Address:b5d7480d-060b-9998-56c6-b9848693bb43} {Suffrage:Voter ID:server-297af9cb-ab45-0290-cf01-fd4970ed2a9e Address:297af9cb-ab45-0290-cf01-fd4970ed2a9e}]" +2024-07-05T13:35:48.218Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: entering follower state: follower="Node at ab67aa8e-194d-20d7-4458-cd7d8969b438 [Follower]" leader-address= leader-id= +2024-07-05T13:35:48.219Z [INFO] server-b5d7480d-060b-9998-56c6-b9848693bb43: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ab67aa8e-194d-20d7-4458-cd7d8969b438 Address:ab67aa8e-194d-20d7-4458-cd7d8969b438} {Suffrage:Voter ID:server-b5d7480d-060b-9998-56c6-b9848693bb43 Address:b5d7480d-060b-9998-56c6-b9848693bb43} {Suffrage:Voter ID:server-297af9cb-ab45-0290-cf01-fd4970ed2a9e Address:297af9cb-ab45-0290-cf01-fd4970ed2a9e}]" +2024-07-05T13:35:48.219Z [INFO] server-b5d7480d-060b-9998-56c6-b9848693bb43: entering follower state: follower="Node at b5d7480d-060b-9998-56c6-b9848693bb43 [Follower]" leader-address= leader-id= +2024-07-05T13:35:48.220Z [INFO] server-297af9cb-ab45-0290-cf01-fd4970ed2a9e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ab67aa8e-194d-20d7-4458-cd7d8969b438 Address:ab67aa8e-194d-20d7-4458-cd7d8969b438} {Suffrage:Voter ID:server-b5d7480d-060b-9998-56c6-b9848693bb43 Address:b5d7480d-060b-9998-56c6-b9848693bb43} {Suffrage:Voter ID:server-297af9cb-ab45-0290-cf01-fd4970ed2a9e Address:297af9cb-ab45-0290-cf01-fd4970ed2a9e}]" +2024-07-05T13:35:48.220Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:48.220Z [INFO] server-297af9cb-ab45-0290-cf01-fd4970ed2a9e: entering follower state: follower="Node at 297af9cb-ab45-0290-cf01-fd4970ed2a9e [Follower]" leader-address= leader-id= +2024-07-05T13:35:48.755Z [WARN] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:48.755Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: entering candidate state: node="Node at ab67aa8e-194d-20d7-4458-cd7d8969b438 [Candidate]" term=2 +2024-07-05T13:35:48.756Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: election won: term=2 tally=2 +2024-07-05T13:35:48.756Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: entering leader state: leader="Node at ab67aa8e-194d-20d7-4458-cd7d8969b438 [Leader]" +2024-07-05T13:35:48.756Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: added peer, starting replication: peer=server-b5d7480d-060b-9998-56c6-b9848693bb43 +2024-07-05T13:35:48.756Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: added peer, starting replication: peer=server-297af9cb-ab45-0290-cf01-fd4970ed2a9e +2024-07-05T13:35:48.758Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: pipelining replication: peer="{Voter server-297af9cb-ab45-0290-cf01-fd4970ed2a9e 297af9cb-ab45-0290-cf01-fd4970ed2a9e}" +2024-07-05T13:35:48.758Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: pipelining replication: peer="{Voter server-b5d7480d-060b-9998-56c6-b9848693bb43 b5d7480d-060b-9998-56c6-b9848693bb43}" +2024-07-05T13:35:49.763Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:48.757586547 +0000 UTC m=+14.753507707 (1 nodes), 536.672322ms from start of poll, 539.840639ms from cluster start. Timeout at 2024-07-05 13:35:49.763674961 +0000 UTC m=+15.759596412, 1.006088705s after stability +2024-07-05T13:35:49.771Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: starting snapshot up to: index=12 +2024-07-05T13:35:49.771Z [INFO] creating new snapshot: path=/tmp/raft3159342172/snapshots/2-12-1720186549771.tmp +2024-07-05T13:35:49.780Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: snapshot complete up to: index=12 +2024-07-05T13:35:49.786Z [INFO] creating new snapshot: path=/tmp/raft3159342172/snapshots/2-23-1720186549786.tmp +2024-07-05T13:35:49.791Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: copied to local snapshot: bytes=71 +2024-07-05T13:35:49.792Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: snapshot restore progress: id=2-23-1720186549786 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:35:49.792Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: restored user snapshot: index=23 +2024-07-05T13:35:49.792Z [ERROR] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: failed to get log: index=23 error="log not found" +2024-07-05T13:35:49.793Z [ERROR] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: failed to get log: index=23 error="log not found" +2024-07-05T13:35:49.793Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: aborting pipeline replication: peer="{Voter server-297af9cb-ab45-0290-cf01-fd4970ed2a9e 297af9cb-ab45-0290-cf01-fd4970ed2a9e}" +2024-07-05T13:35:49.793Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: aborting pipeline replication: peer="{Voter server-b5d7480d-060b-9998-56c6-b9848693bb43 b5d7480d-060b-9998-56c6-b9848693bb43}" +2024-07-05T13:35:49.799Z [ERROR] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: failed to get log: index=23 error="log not found" +2024-07-05T13:35:49.800Z [ERROR] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: failed to get log: index=23 error="log not found" +2024-07-05T13:35:49.802Z [INFO] creating new snapshot: path=/tmp/raft1871483886/snapshots/2-23-1720186549802.tmp +2024-07-05T13:35:49.802Z [INFO] creating new snapshot: path=/tmp/raft2617308453/snapshots/2-23-1720186549802.tmp +2024-07-05T13:35:49.804Z [INFO] server-b5d7480d-060b-9998-56c6-b9848693bb43: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:35:49.804Z [INFO] server-297af9cb-ab45-0290-cf01-fd4970ed2a9e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:35:49.806Z [INFO] server-b5d7480d-060b-9998-56c6-b9848693bb43: copied to local snapshot: bytes=71 +2024-07-05T13:35:49.807Z [INFO] server-297af9cb-ab45-0290-cf01-fd4970ed2a9e: copied to local snapshot: bytes=71 +2024-07-05T13:35:49.807Z [INFO] server-b5d7480d-060b-9998-56c6-b9848693bb43: snapshot restore progress: id=2-23-1720186549802 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:35:49.808Z [INFO] server-b5d7480d-060b-9998-56c6-b9848693bb43: Installed remote snapshot +2024-07-05T13:35:49.808Z [INFO] server-297af9cb-ab45-0290-cf01-fd4970ed2a9e: snapshot restore progress: id=2-23-1720186549802 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:35:49.808Z [WARN] server-b5d7480d-060b-9998-56c6-b9848693bb43: skipping application of old log: index=23 +2024-07-05T13:35:49.809Z [INFO] server-297af9cb-ab45-0290-cf01-fd4970ed2a9e: Installed remote snapshot +2024-07-05T13:35:49.809Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: pipelining replication: peer="{Voter server-b5d7480d-060b-9998-56c6-b9848693bb43 b5d7480d-060b-9998-56c6-b9848693bb43}" +2024-07-05T13:35:49.809Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:35:49.810Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: pipelining replication: peer="{Voter server-297af9cb-ab45-0290-cf01-fd4970ed2a9e 297af9cb-ab45-0290-cf01-fd4970ed2a9e}" +2024-07-05T13:35:50.815Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:35:49.809856909 +0000 UTC m=+15.805778360 (2 nodes), 8.458µs from start of poll, 1.592111292s from cluster start. Timeout at 2024-07-05 13:35:50.815637912 +0000 UTC m=+16.811559363, 1.005781003s after stability +2024-07-05T13:35:50.831Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: aborting pipeline replication: peer="{Voter server-297af9cb-ab45-0290-cf01-fd4970ed2a9e 297af9cb-ab45-0290-cf01-fd4970ed2a9e}" +2024-07-05T13:35:50.831Z [INFO] server-ab67aa8e-194d-20d7-4458-cd7d8969b438: aborting pipeline replication: peer="{Voter server-b5d7480d-060b-9998-56c6-b9848693bb43 b5d7480d-060b-9998-56c6-b9848693bb43}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false -2024-07-05T13:29:35.350Z [INFO] server-61abf79c-8dd5-f1d8-1e93-55dc056e9888: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-61abf79c-8dd5-f1d8-1e93-55dc056e9888 Address:61abf79c-8dd5-f1d8-1e93-55dc056e9888} {Suffrage:Voter ID:server-1a15750f-3e05-64f1-7c76-087aad798f5c Address:1a15750f-3e05-64f1-7c76-087aad798f5c} {Suffrage:Voter ID:server-4169d41a-4eb6-3328-f4a2-594e5c662bd3 Address:4169d41a-4eb6-3328-f4a2-594e5c662bd3}]" -2024-07-05T13:29:35.350Z [INFO] server-1a15750f-3e05-64f1-7c76-087aad798f5c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-61abf79c-8dd5-f1d8-1e93-55dc056e9888 Address:61abf79c-8dd5-f1d8-1e93-55dc056e9888} {Suffrage:Voter ID:server-1a15750f-3e05-64f1-7c76-087aad798f5c Address:1a15750f-3e05-64f1-7c76-087aad798f5c} {Suffrage:Voter ID:server-4169d41a-4eb6-3328-f4a2-594e5c662bd3 Address:4169d41a-4eb6-3328-f4a2-594e5c662bd3}]" -2024-07-05T13:29:35.350Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-61abf79c-8dd5-f1d8-1e93-55dc056e9888 Address:61abf79c-8dd5-f1d8-1e93-55dc056e9888} {Suffrage:Voter ID:server-1a15750f-3e05-64f1-7c76-087aad798f5c Address:1a15750f-3e05-64f1-7c76-087aad798f5c} {Suffrage:Voter ID:server-4169d41a-4eb6-3328-f4a2-594e5c662bd3 Address:4169d41a-4eb6-3328-f4a2-594e5c662bd3}]" -2024-07-05T13:29:35.351Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:35.352Z [INFO] server-1a15750f-3e05-64f1-7c76-087aad798f5c: entering follower state: follower="Node at 1a15750f-3e05-64f1-7c76-087aad798f5c [Follower]" leader-address= leader-id= -2024-07-05T13:29:35.352Z [INFO] server-61abf79c-8dd5-f1d8-1e93-55dc056e9888: entering follower state: follower="Node at 61abf79c-8dd5-f1d8-1e93-55dc056e9888 [Follower]" leader-address= leader-id= -2024-07-05T13:29:35.352Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: entering follower state: follower="Node at 4169d41a-4eb6-3328-f4a2-594e5c662bd3 [Follower]" leader-address= leader-id= -2024-07-05T13:29:36.219Z [WARN] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:36.219Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: entering candidate state: node="Node at 4169d41a-4eb6-3328-f4a2-594e5c662bd3 [Candidate]" term=2 -2024-07-05T13:29:36.220Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: election won: term=2 tally=2 -2024-07-05T13:29:36.220Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: entering leader state: leader="Node at 4169d41a-4eb6-3328-f4a2-594e5c662bd3 [Leader]" -2024-07-05T13:29:36.220Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: added peer, starting replication: peer=server-61abf79c-8dd5-f1d8-1e93-55dc056e9888 -2024-07-05T13:29:36.220Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: added peer, starting replication: peer=server-1a15750f-3e05-64f1-7c76-087aad798f5c -2024-07-05T13:29:36.220Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: pipelining replication: peer="{Voter server-1a15750f-3e05-64f1-7c76-087aad798f5c 1a15750f-3e05-64f1-7c76-087aad798f5c}" -2024-07-05T13:29:36.220Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: pipelining replication: peer="{Voter server-61abf79c-8dd5-f1d8-1e93-55dc056e9888 61abf79c-8dd5-f1d8-1e93-55dc056e9888}" -2024-07-05T13:29:37.226Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:36.220737044 +0000 UTC m=+16.990647719 (1 nodes), 869.522536ms from start of poll, 870.734254ms from cluster start. Timeout at 2024-07-05 13:29:37.226329167 +0000 UTC m=+17.996239842, 1.005592123s after stability -2024-07-05T13:29:37.228Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: starting snapshot up to: index=12 -2024-07-05T13:29:37.228Z [INFO] creating new snapshot: path=/tmp/raft2524423408/snapshots/2-12-1720186177228.tmp -2024-07-05T13:29:37.229Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: snapshot complete up to: index=12 -2024-07-05T13:29:37.232Z [INFO] creating new snapshot: path=/tmp/raft2524423408/snapshots/2-23-1720186177232.tmp -2024-07-05T13:29:37.233Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: copied to local snapshot: bytes=71 -2024-07-05T13:29:37.233Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: snapshot restore progress: id=2-23-1720186177232 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:37.233Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: removing all old logs from log store -2024-07-05T13:29:37.233Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: compacting logs: from=1 to=22 -2024-07-05T13:29:37.233Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: restored user snapshot: index=23 -2024-07-05T13:29:37.233Z [ERROR] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: failed to get log: index=22 error="log not found" -2024-07-05T13:29:37.233Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: aborting pipeline replication: peer="{Voter server-1a15750f-3e05-64f1-7c76-087aad798f5c 1a15750f-3e05-64f1-7c76-087aad798f5c}" -2024-07-05T13:29:37.234Z [ERROR] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: failed to get log: index=22 error="log not found" -2024-07-05T13:29:37.234Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: aborting pipeline replication: peer="{Voter server-61abf79c-8dd5-f1d8-1e93-55dc056e9888 61abf79c-8dd5-f1d8-1e93-55dc056e9888}" -2024-07-05T13:29:37.239Z [ERROR] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: failed to get log: index=22 error="log not found" -2024-07-05T13:29:37.240Z [ERROR] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: failed to get log: index=22 error="log not found" -2024-07-05T13:29:37.241Z [INFO] creating new snapshot: path=/tmp/raft2000203495/snapshots/2-23-1720186177241.tmp -2024-07-05T13:29:37.242Z [INFO] creating new snapshot: path=/tmp/raft3374663992/snapshots/2-23-1720186177242.tmp -2024-07-05T13:29:37.243Z [INFO] server-61abf79c-8dd5-f1d8-1e93-55dc056e9888: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:37.244Z [INFO] server-1a15750f-3e05-64f1-7c76-087aad798f5c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:37.244Z [INFO] server-61abf79c-8dd5-f1d8-1e93-55dc056e9888: copied to local snapshot: bytes=71 -2024-07-05T13:29:37.245Z [INFO] server-1a15750f-3e05-64f1-7c76-087aad798f5c: copied to local snapshot: bytes=71 -2024-07-05T13:29:37.245Z [INFO] server-61abf79c-8dd5-f1d8-1e93-55dc056e9888: snapshot restore progress: id=2-23-1720186177241 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:37.245Z [INFO] server-61abf79c-8dd5-f1d8-1e93-55dc056e9888: removing all old logs from log store -2024-07-05T13:29:37.245Z [INFO] server-61abf79c-8dd5-f1d8-1e93-55dc056e9888: compacting logs: from=1 to=22 -2024-07-05T13:29:37.245Z [INFO] server-61abf79c-8dd5-f1d8-1e93-55dc056e9888: Installed remote snapshot -2024-07-05T13:29:37.245Z [WARN] server-61abf79c-8dd5-f1d8-1e93-55dc056e9888: skipping application of old log: index=22 -2024-07-05T13:29:37.245Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: pipelining replication: peer="{Voter server-61abf79c-8dd5-f1d8-1e93-55dc056e9888 61abf79c-8dd5-f1d8-1e93-55dc056e9888}" -2024-07-05T13:29:37.245Z [INFO] server-1a15750f-3e05-64f1-7c76-087aad798f5c: snapshot restore progress: id=2-23-1720186177242 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:37.245Z [INFO] server-1a15750f-3e05-64f1-7c76-087aad798f5c: removing all old logs from log store -2024-07-05T13:29:37.245Z [INFO] server-1a15750f-3e05-64f1-7c76-087aad798f5c: compacting logs: from=1 to=22 -2024-07-05T13:29:37.245Z [INFO] server-1a15750f-3e05-64f1-7c76-087aad798f5c: Installed remote snapshot -2024-07-05T13:29:37.245Z [WARN] server-1a15750f-3e05-64f1-7c76-087aad798f5c: skipping application of old log: index=22 -2024-07-05T13:29:37.246Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: pipelining replication: peer="{Voter server-1a15750f-3e05-64f1-7c76-087aad798f5c 1a15750f-3e05-64f1-7c76-087aad798f5c}" -2024-07-05T13:29:37.246Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:29:38.251Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:29:37.246245348 +0000 UTC m=+18.016156023 (2 nodes), 1.719µs from start of poll, 1.896242558s from cluster start. Timeout at 2024-07-05 13:29:38.251748314 +0000 UTC m=+19.021658989, 1.005502966s after stability -2024-07-05T13:29:38.265Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: aborting pipeline replication: peer="{Voter server-1a15750f-3e05-64f1-7c76-087aad798f5c 1a15750f-3e05-64f1-7c76-087aad798f5c}" -2024-07-05T13:29:38.265Z [INFO] server-4169d41a-4eb6-3328-f4a2-594e5c662bd3: aborting pipeline replication: peer="{Voter server-61abf79c-8dd5-f1d8-1e93-55dc056e9888 61abf79c-8dd5-f1d8-1e93-55dc056e9888}" +2024-07-05T13:35:50.843Z [INFO] server-b0d182e2-8339-691b-0ac3-baab4c261678: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b0d182e2-8339-691b-0ac3-baab4c261678 Address:b0d182e2-8339-691b-0ac3-baab4c261678} {Suffrage:Voter ID:server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6 Address:bbc89f5a-2dea-5bb3-c000-22f68ae70cb6} {Suffrage:Voter ID:server-484f95bb-68bb-3140-6acf-439016280bbe Address:484f95bb-68bb-3140-6acf-439016280bbe}]" +2024-07-05T13:35:50.844Z [INFO] server-b0d182e2-8339-691b-0ac3-baab4c261678: entering follower state: follower="Node at b0d182e2-8339-691b-0ac3-baab4c261678 [Follower]" leader-address= leader-id= +2024-07-05T13:35:50.844Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b0d182e2-8339-691b-0ac3-baab4c261678 Address:b0d182e2-8339-691b-0ac3-baab4c261678} {Suffrage:Voter ID:server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6 Address:bbc89f5a-2dea-5bb3-c000-22f68ae70cb6} {Suffrage:Voter ID:server-484f95bb-68bb-3140-6acf-439016280bbe Address:484f95bb-68bb-3140-6acf-439016280bbe}]" +2024-07-05T13:35:50.844Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: entering follower state: follower="Node at bbc89f5a-2dea-5bb3-c000-22f68ae70cb6 [Follower]" leader-address= leader-id= +2024-07-05T13:35:50.845Z [INFO] server-484f95bb-68bb-3140-6acf-439016280bbe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b0d182e2-8339-691b-0ac3-baab4c261678 Address:b0d182e2-8339-691b-0ac3-baab4c261678} {Suffrage:Voter ID:server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6 Address:bbc89f5a-2dea-5bb3-c000-22f68ae70cb6} {Suffrage:Voter ID:server-484f95bb-68bb-3140-6acf-439016280bbe Address:484f95bb-68bb-3140-6acf-439016280bbe}]" +2024-07-05T13:35:50.845Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:50.845Z [INFO] server-484f95bb-68bb-3140-6acf-439016280bbe: entering follower state: follower="Node at 484f95bb-68bb-3140-6acf-439016280bbe [Follower]" leader-address= leader-id= +2024-07-05T13:35:51.352Z [WARN] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:51.352Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: entering candidate state: node="Node at bbc89f5a-2dea-5bb3-c000-22f68ae70cb6 [Candidate]" term=2 +2024-07-05T13:35:51.352Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: election won: term=2 tally=2 +2024-07-05T13:35:51.353Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: entering leader state: leader="Node at bbc89f5a-2dea-5bb3-c000-22f68ae70cb6 [Leader]" +2024-07-05T13:35:51.353Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: added peer, starting replication: peer=server-b0d182e2-8339-691b-0ac3-baab4c261678 +2024-07-05T13:35:51.353Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: added peer, starting replication: peer=server-484f95bb-68bb-3140-6acf-439016280bbe +2024-07-05T13:35:51.354Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: pipelining replication: peer="{Voter server-b0d182e2-8339-691b-0ac3-baab4c261678 b0d182e2-8339-691b-0ac3-baab4c261678}" +2024-07-05T13:35:51.354Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: pipelining replication: peer="{Voter server-484f95bb-68bb-3140-6acf-439016280bbe 484f95bb-68bb-3140-6acf-439016280bbe}" +2024-07-05T13:35:52.359Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:51.353320257 +0000 UTC m=+17.349241708 (1 nodes), 507.862598ms from start of poll, 509.974226ms from cluster start. Timeout at 2024-07-05 13:35:52.358881347 +0000 UTC m=+18.354802799, 1.005561091s after stability +2024-07-05T13:35:52.363Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: starting snapshot up to: index=12 +2024-07-05T13:35:52.363Z [INFO] creating new snapshot: path=/tmp/raft4054322673/snapshots/2-12-1720186552363.tmp +2024-07-05T13:35:52.367Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: snapshot complete up to: index=12 +2024-07-05T13:35:52.374Z [INFO] creating new snapshot: path=/tmp/raft4054322673/snapshots/2-23-1720186552374.tmp +2024-07-05T13:35:52.377Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: copied to local snapshot: bytes=71 +2024-07-05T13:35:52.379Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: snapshot restore progress: id=2-23-1720186552374 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:35:52.379Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: removing all old logs from log store +2024-07-05T13:35:52.379Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: compacting logs: from=1 to=22 +2024-07-05T13:35:52.379Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: restored user snapshot: index=23 +2024-07-05T13:35:52.380Z [ERROR] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: failed to get log: index=22 error="log not found" +2024-07-05T13:35:52.380Z [ERROR] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: failed to get log: index=22 error="log not found" +2024-07-05T13:35:52.380Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: aborting pipeline replication: peer="{Voter server-484f95bb-68bb-3140-6acf-439016280bbe 484f95bb-68bb-3140-6acf-439016280bbe}" +2024-07-05T13:35:52.380Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: aborting pipeline replication: peer="{Voter server-b0d182e2-8339-691b-0ac3-baab4c261678 b0d182e2-8339-691b-0ac3-baab4c261678}" +2024-07-05T13:35:52.387Z [ERROR] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: failed to get log: index=22 error="log not found" +2024-07-05T13:35:52.389Z [ERROR] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: failed to get log: index=22 error="log not found" +2024-07-05T13:35:52.390Z [INFO] creating new snapshot: path=/tmp/raft2658156407/snapshots/2-23-1720186552390.tmp +2024-07-05T13:35:52.392Z [INFO] server-b0d182e2-8339-691b-0ac3-baab4c261678: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:35:52.392Z [INFO] creating new snapshot: path=/tmp/raft2529733354/snapshots/2-23-1720186552392.tmp +2024-07-05T13:35:52.394Z [INFO] server-484f95bb-68bb-3140-6acf-439016280bbe: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:35:52.394Z [INFO] server-b0d182e2-8339-691b-0ac3-baab4c261678: copied to local snapshot: bytes=71 +2024-07-05T13:35:52.396Z [INFO] server-484f95bb-68bb-3140-6acf-439016280bbe: copied to local snapshot: bytes=71 +2024-07-05T13:35:52.396Z [INFO] server-b0d182e2-8339-691b-0ac3-baab4c261678: snapshot restore progress: id=2-23-1720186552390 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:35:52.396Z [INFO] server-b0d182e2-8339-691b-0ac3-baab4c261678: removing all old logs from log store +2024-07-05T13:35:52.396Z [INFO] server-b0d182e2-8339-691b-0ac3-baab4c261678: compacting logs: from=1 to=22 +2024-07-05T13:35:52.396Z [INFO] server-b0d182e2-8339-691b-0ac3-baab4c261678: Installed remote snapshot +2024-07-05T13:35:52.397Z [INFO] server-484f95bb-68bb-3140-6acf-439016280bbe: snapshot restore progress: id=2-23-1720186552392 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:35:52.397Z [INFO] server-484f95bb-68bb-3140-6acf-439016280bbe: removing all old logs from log store +2024-07-05T13:35:52.397Z [INFO] server-484f95bb-68bb-3140-6acf-439016280bbe: compacting logs: from=1 to=22 +2024-07-05T13:35:52.397Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: pipelining replication: peer="{Voter server-b0d182e2-8339-691b-0ac3-baab4c261678 b0d182e2-8339-691b-0ac3-baab4c261678}" +2024-07-05T13:35:52.397Z [INFO] server-484f95bb-68bb-3140-6acf-439016280bbe: Installed remote snapshot +2024-07-05T13:35:52.398Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:35:52.398Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: pipelining replication: peer="{Voter server-484f95bb-68bb-3140-6acf-439016280bbe 484f95bb-68bb-3140-6acf-439016280bbe}" +2024-07-05T13:35:53.403Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:35:52.398230837 +0000 UTC m=+18.394152288 (2 nodes), 7.291µs from start of poll, 1.554884806s from cluster start. Timeout at 2024-07-05 13:35:53.403601764 +0000 UTC m=+19.399523216, 1.005370928s after stability +2024-07-05T13:35:53.419Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: aborting pipeline replication: peer="{Voter server-484f95bb-68bb-3140-6acf-439016280bbe 484f95bb-68bb-3140-6acf-439016280bbe}" +2024-07-05T13:35:53.419Z [INFO] server-bbc89f5a-2dea-5bb3-c000-22f68ae70cb6: aborting pipeline replication: peer="{Voter server-b0d182e2-8339-691b-0ac3-baab4c261678 b0d182e2-8339-691b-0ac3-baab4c261678}" === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true -2024-07-05T13:29:38.268Z [INFO] server-0c5a2f98-1fda-115b-b0bb-300d8ef74abd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c5a2f98-1fda-115b-b0bb-300d8ef74abd Address:0c5a2f98-1fda-115b-b0bb-300d8ef74abd} {Suffrage:Voter ID:server-e220e863-271d-94c0-27a9-4aff8d10db47 Address:e220e863-271d-94c0-27a9-4aff8d10db47} {Suffrage:Voter ID:server-60f9cd73-ca1e-b794-2762-764fd10bdb8d Address:60f9cd73-ca1e-b794-2762-764fd10bdb8d}]" -2024-07-05T13:29:38.269Z [INFO] server-e220e863-271d-94c0-27a9-4aff8d10db47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c5a2f98-1fda-115b-b0bb-300d8ef74abd Address:0c5a2f98-1fda-115b-b0bb-300d8ef74abd} {Suffrage:Voter ID:server-e220e863-271d-94c0-27a9-4aff8d10db47 Address:e220e863-271d-94c0-27a9-4aff8d10db47} {Suffrage:Voter ID:server-60f9cd73-ca1e-b794-2762-764fd10bdb8d Address:60f9cd73-ca1e-b794-2762-764fd10bdb8d}]" -2024-07-05T13:29:38.269Z [INFO] server-60f9cd73-ca1e-b794-2762-764fd10bdb8d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c5a2f98-1fda-115b-b0bb-300d8ef74abd Address:0c5a2f98-1fda-115b-b0bb-300d8ef74abd} {Suffrage:Voter ID:server-e220e863-271d-94c0-27a9-4aff8d10db47 Address:e220e863-271d-94c0-27a9-4aff8d10db47} {Suffrage:Voter ID:server-60f9cd73-ca1e-b794-2762-764fd10bdb8d Address:60f9cd73-ca1e-b794-2762-764fd10bdb8d}]" -2024-07-05T13:29:38.269Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:38.269Z [INFO] server-0c5a2f98-1fda-115b-b0bb-300d8ef74abd: entering follower state: follower="Node at 0c5a2f98-1fda-115b-b0bb-300d8ef74abd [Follower]" leader-address= leader-id= -2024-07-05T13:29:38.269Z [INFO] server-e220e863-271d-94c0-27a9-4aff8d10db47: entering follower state: follower="Node at e220e863-271d-94c0-27a9-4aff8d10db47 [Follower]" leader-address= leader-id= -2024-07-05T13:29:38.269Z [INFO] server-60f9cd73-ca1e-b794-2762-764fd10bdb8d: entering follower state: follower="Node at 60f9cd73-ca1e-b794-2762-764fd10bdb8d [Follower]" leader-address= leader-id= -2024-07-05T13:29:39.036Z [WARN] server-e220e863-271d-94c0-27a9-4aff8d10db47: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:39.036Z [INFO] server-e220e863-271d-94c0-27a9-4aff8d10db47: entering candidate state: node="Node at e220e863-271d-94c0-27a9-4aff8d10db47 [Candidate]" term=2 -2024-07-05T13:29:39.036Z [INFO] server-e220e863-271d-94c0-27a9-4aff8d10db47: election won: term=2 tally=2 -2024-07-05T13:29:39.036Z [INFO] server-e220e863-271d-94c0-27a9-4aff8d10db47: entering leader state: leader="Node at e220e863-271d-94c0-27a9-4aff8d10db47 [Leader]" -2024-07-05T13:29:39.036Z [INFO] server-e220e863-271d-94c0-27a9-4aff8d10db47: added peer, starting replication: peer=server-0c5a2f98-1fda-115b-b0bb-300d8ef74abd -2024-07-05T13:29:39.036Z [INFO] server-e220e863-271d-94c0-27a9-4aff8d10db47: added peer, starting replication: peer=server-60f9cd73-ca1e-b794-2762-764fd10bdb8d -2024-07-05T13:29:39.036Z [INFO] server-e220e863-271d-94c0-27a9-4aff8d10db47: pipelining replication: peer="{Voter server-60f9cd73-ca1e-b794-2762-764fd10bdb8d 60f9cd73-ca1e-b794-2762-764fd10bdb8d}" -2024-07-05T13:29:39.036Z [INFO] server-e220e863-271d-94c0-27a9-4aff8d10db47: pipelining replication: peer="{Voter server-0c5a2f98-1fda-115b-b0bb-300d8ef74abd 0c5a2f98-1fda-115b-b0bb-300d8ef74abd}" -2024-07-05T13:29:39.037Z [WARN] server-0c5a2f98-1fda-115b-b0bb-300d8ef74abd: rejecting vote request since our last term is greater: candidate=e220e863-271d-94c0-27a9-4aff8d10db47 last-term=2 last-candidate-term=1 -2024-07-05T13:29:40.042Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:39.037230399 +0000 UTC m=+19.807141074 (1 nodes), 767.967179ms from start of poll, 768.732312ms from cluster start. Timeout at 2024-07-05 13:29:40.042759924 +0000 UTC m=+20.812670599, 1.005529525s after stability -2024-07-05T13:29:40.044Z [INFO] server-e220e863-271d-94c0-27a9-4aff8d10db47: starting snapshot up to: index=12 -2024-07-05T13:29:40.044Z [INFO] creating new snapshot: path=/tmp/raft798463498/snapshots/2-12-1720186180044.tmp -2024-07-05T13:29:40.045Z [INFO] server-e220e863-271d-94c0-27a9-4aff8d10db47: snapshot complete up to: index=12 -2024-07-05T13:29:40.050Z [INFO] server-e3394e2a-e781-cbc2-69d8-964bdb465a9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3394e2a-e781-cbc2-69d8-964bdb465a9d Address:e3394e2a-e781-cbc2-69d8-964bdb465a9d} {Suffrage:Voter ID:server-25a57ff3-bc21-d4ec-e92e-acc7485d6a54 Address:25a57ff3-bc21-d4ec-e92e-acc7485d6a54} {Suffrage:Voter ID:server-476e4ba0-a53a-753c-3bd8-dfcde18fd777 Address:476e4ba0-a53a-753c-3bd8-dfcde18fd777}]" -2024-07-05T13:29:40.050Z [INFO] server-25a57ff3-bc21-d4ec-e92e-acc7485d6a54: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3394e2a-e781-cbc2-69d8-964bdb465a9d Address:e3394e2a-e781-cbc2-69d8-964bdb465a9d} {Suffrage:Voter ID:server-25a57ff3-bc21-d4ec-e92e-acc7485d6a54 Address:25a57ff3-bc21-d4ec-e92e-acc7485d6a54} {Suffrage:Voter ID:server-476e4ba0-a53a-753c-3bd8-dfcde18fd777 Address:476e4ba0-a53a-753c-3bd8-dfcde18fd777}]" -2024-07-05T13:29:40.050Z [INFO] server-e3394e2a-e781-cbc2-69d8-964bdb465a9d: entering follower state: follower="Node at e3394e2a-e781-cbc2-69d8-964bdb465a9d [Follower]" leader-address= leader-id= -2024-07-05T13:29:40.050Z [INFO] server-25a57ff3-bc21-d4ec-e92e-acc7485d6a54: entering follower state: follower="Node at 25a57ff3-bc21-d4ec-e92e-acc7485d6a54 [Follower]" leader-address= leader-id= -2024-07-05T13:29:40.051Z [INFO] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3394e2a-e781-cbc2-69d8-964bdb465a9d Address:e3394e2a-e781-cbc2-69d8-964bdb465a9d} {Suffrage:Voter ID:server-25a57ff3-bc21-d4ec-e92e-acc7485d6a54 Address:25a57ff3-bc21-d4ec-e92e-acc7485d6a54} {Suffrage:Voter ID:server-476e4ba0-a53a-753c-3bd8-dfcde18fd777 Address:476e4ba0-a53a-753c-3bd8-dfcde18fd777}]" -2024-07-05T13:29:40.051Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:40.051Z [INFO] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: entering follower state: follower="Node at 476e4ba0-a53a-753c-3bd8-dfcde18fd777 [Follower]" leader-address= leader-id= -2024-07-05T13:29:40.768Z [WARN] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:40.768Z [INFO] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: entering candidate state: node="Node at 476e4ba0-a53a-753c-3bd8-dfcde18fd777 [Candidate]" term=2 -2024-07-05T13:29:40.768Z [INFO] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: election won: term=2 tally=2 -2024-07-05T13:29:40.768Z [INFO] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: entering leader state: leader="Node at 476e4ba0-a53a-753c-3bd8-dfcde18fd777 [Leader]" -2024-07-05T13:29:40.768Z [INFO] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: added peer, starting replication: peer=server-e3394e2a-e781-cbc2-69d8-964bdb465a9d -2024-07-05T13:29:40.768Z [INFO] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: added peer, starting replication: peer=server-25a57ff3-bc21-d4ec-e92e-acc7485d6a54 -2024-07-05T13:29:40.768Z [INFO] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: pipelining replication: peer="{Voter server-25a57ff3-bc21-d4ec-e92e-acc7485d6a54 25a57ff3-bc21-d4ec-e92e-acc7485d6a54}" -2024-07-05T13:29:40.769Z [INFO] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: pipelining replication: peer="{Voter server-e3394e2a-e781-cbc2-69d8-964bdb465a9d e3394e2a-e781-cbc2-69d8-964bdb465a9d}" -2024-07-05T13:29:41.774Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:40.769380753 +0000 UTC m=+21.539291428 (1 nodes), 718.148046ms from start of poll, 719.529918ms from cluster start. Timeout at 2024-07-05 13:29:41.774500012 +0000 UTC m=+22.544410687, 1.005119259s after stability -2024-07-05T13:29:41.775Z [INFO] creating new snapshot: path=/tmp/raft45978922/snapshots/2-13-1720186181775.tmp -2024-07-05T13:29:41.776Z [INFO] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: copied to local snapshot: bytes=71 -2024-07-05T13:29:41.776Z [INFO] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: snapshot restore progress: id=2-13-1720186181775 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:41.776Z [INFO] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: restored user snapshot: index=13 -2024-07-05T13:29:41.776Z [ERROR] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: failed to get log: index=3 error="log not found" -2024-07-05T13:29:41.776Z [INFO] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: aborting pipeline replication: peer="{Voter server-25a57ff3-bc21-d4ec-e92e-acc7485d6a54 25a57ff3-bc21-d4ec-e92e-acc7485d6a54}" -2024-07-05T13:29:41.776Z [ERROR] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: failed to get log: index=3 error="log not found" -2024-07-05T13:29:41.776Z [INFO] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: aborting pipeline replication: peer="{Voter server-e3394e2a-e781-cbc2-69d8-964bdb465a9d e3394e2a-e781-cbc2-69d8-964bdb465a9d}" -2024-07-05T13:29:41.786Z [ERROR] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: failed to get log: index=3 error="log not found" -2024-07-05T13:29:41.786Z [INFO] creating new snapshot: path=/tmp/raft2703585316/snapshots/2-13-1720186181786.tmp -2024-07-05T13:29:41.787Z [INFO] server-e3394e2a-e781-cbc2-69d8-964bdb465a9d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:41.787Z [INFO] server-e3394e2a-e781-cbc2-69d8-964bdb465a9d: copied to local snapshot: bytes=71 -2024-07-05T13:29:41.788Z [INFO] server-e3394e2a-e781-cbc2-69d8-964bdb465a9d: snapshot restore progress: id=2-13-1720186181786 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:41.788Z [INFO] server-e3394e2a-e781-cbc2-69d8-964bdb465a9d: Installed remote snapshot -2024-07-05T13:29:41.788Z [INFO] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: pipelining replication: peer="{Voter server-e3394e2a-e781-cbc2-69d8-964bdb465a9d e3394e2a-e781-cbc2-69d8-964bdb465a9d}" -2024-07-05T13:29:41.788Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:29:41.788Z [ERROR] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: failed to get log: index=3 error="log not found" -2024-07-05T13:29:41.789Z [INFO] creating new snapshot: path=/tmp/raft1711569820/snapshots/2-13-1720186181789.tmp -2024-07-05T13:29:41.789Z [INFO] server-25a57ff3-bc21-d4ec-e92e-acc7485d6a54: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:41.790Z [INFO] server-25a57ff3-bc21-d4ec-e92e-acc7485d6a54: copied to local snapshot: bytes=71 -2024-07-05T13:29:41.790Z [INFO] server-25a57ff3-bc21-d4ec-e92e-acc7485d6a54: snapshot restore progress: id=2-13-1720186181789 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:41.790Z [INFO] server-25a57ff3-bc21-d4ec-e92e-acc7485d6a54: Installed remote snapshot -2024-07-05T13:29:41.791Z [INFO] server-476e4ba0-a53a-753c-3bd8-dfcde18fd777: pipelining replication: peer="{Voter server-25a57ff3-bc21-d4ec-e92e-acc7485d6a54 25a57ff3-bc21-d4ec-e92e-acc7485d6a54}" -2024-07-05T13:29:42.794Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:29:41.788767552 +0000 UTC m=+22.558678227 (2 nodes), 1.52µs from start of poll, 1.738916717s from cluster start. Timeout at 2024-07-05 13:29:42.79450211 +0000 UTC m=+23.564412785, 1.005734558s after stability -2024-07-05T13:29:42.804Z [INFO] server-e220e863-271d-94c0-27a9-4aff8d10db47: aborting pipeline replication: peer="{Voter server-60f9cd73-ca1e-b794-2762-764fd10bdb8d 60f9cd73-ca1e-b794-2762-764fd10bdb8d}" -2024-07-05T13:29:42.811Z [INFO] server-e220e863-271d-94c0-27a9-4aff8d10db47: aborting pipeline replication: peer="{Voter server-0c5a2f98-1fda-115b-b0bb-300d8ef74abd 0c5a2f98-1fda-115b-b0bb-300d8ef74abd}" +2024-07-05T13:35:53.434Z [INFO] server-e0719c68-24ef-5287-6063-394c93632913: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0719c68-24ef-5287-6063-394c93632913 Address:e0719c68-24ef-5287-6063-394c93632913} {Suffrage:Voter ID:server-b826c733-e625-48cd-a25a-c753aab03365 Address:b826c733-e625-48cd-a25a-c753aab03365} {Suffrage:Voter ID:server-e10b353c-f776-a492-b0ac-a723a30b6647 Address:e10b353c-f776-a492-b0ac-a723a30b6647}]" +2024-07-05T13:35:53.434Z [INFO] server-e0719c68-24ef-5287-6063-394c93632913: entering follower state: follower="Node at e0719c68-24ef-5287-6063-394c93632913 [Follower]" leader-address= leader-id= +2024-07-05T13:35:53.435Z [INFO] server-b826c733-e625-48cd-a25a-c753aab03365: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0719c68-24ef-5287-6063-394c93632913 Address:e0719c68-24ef-5287-6063-394c93632913} {Suffrage:Voter ID:server-b826c733-e625-48cd-a25a-c753aab03365 Address:b826c733-e625-48cd-a25a-c753aab03365} {Suffrage:Voter ID:server-e10b353c-f776-a492-b0ac-a723a30b6647 Address:e10b353c-f776-a492-b0ac-a723a30b6647}]" +2024-07-05T13:35:53.435Z [INFO] server-b826c733-e625-48cd-a25a-c753aab03365: entering follower state: follower="Node at b826c733-e625-48cd-a25a-c753aab03365 [Follower]" leader-address= leader-id= +2024-07-05T13:35:53.436Z [INFO] server-e10b353c-f776-a492-b0ac-a723a30b6647: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0719c68-24ef-5287-6063-394c93632913 Address:e0719c68-24ef-5287-6063-394c93632913} {Suffrage:Voter ID:server-b826c733-e625-48cd-a25a-c753aab03365 Address:b826c733-e625-48cd-a25a-c753aab03365} {Suffrage:Voter ID:server-e10b353c-f776-a492-b0ac-a723a30b6647 Address:e10b353c-f776-a492-b0ac-a723a30b6647}]" +2024-07-05T13:35:53.436Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:53.436Z [INFO] server-e10b353c-f776-a492-b0ac-a723a30b6647: entering follower state: follower="Node at e10b353c-f776-a492-b0ac-a723a30b6647 [Follower]" leader-address= leader-id= +2024-07-05T13:35:54.235Z [WARN] server-e10b353c-f776-a492-b0ac-a723a30b6647: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:54.235Z [INFO] server-e10b353c-f776-a492-b0ac-a723a30b6647: entering candidate state: node="Node at e10b353c-f776-a492-b0ac-a723a30b6647 [Candidate]" term=2 +2024-07-05T13:35:54.235Z [INFO] server-e10b353c-f776-a492-b0ac-a723a30b6647: election won: term=2 tally=2 +2024-07-05T13:35:54.235Z [INFO] server-e10b353c-f776-a492-b0ac-a723a30b6647: entering leader state: leader="Node at e10b353c-f776-a492-b0ac-a723a30b6647 [Leader]" +2024-07-05T13:35:54.236Z [INFO] server-e10b353c-f776-a492-b0ac-a723a30b6647: added peer, starting replication: peer=server-e0719c68-24ef-5287-6063-394c93632913 +2024-07-05T13:35:54.236Z [INFO] server-e10b353c-f776-a492-b0ac-a723a30b6647: added peer, starting replication: peer=server-b826c733-e625-48cd-a25a-c753aab03365 +2024-07-05T13:35:54.236Z [INFO] server-e10b353c-f776-a492-b0ac-a723a30b6647: pipelining replication: peer="{Voter server-e0719c68-24ef-5287-6063-394c93632913 e0719c68-24ef-5287-6063-394c93632913}" +2024-07-05T13:35:54.236Z [INFO] server-e10b353c-f776-a492-b0ac-a723a30b6647: pipelining replication: peer="{Voter server-b826c733-e625-48cd-a25a-c753aab03365 b826c733-e625-48cd-a25a-c753aab03365}" +2024-07-05T13:35:55.242Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:54.236419718 +0000 UTC m=+20.232340878 (1 nodes), 799.926138ms from start of poll, 802.771294ms from cluster start. Timeout at 2024-07-05 13:35:55.242230763 +0000 UTC m=+21.238152214, 1.005811336s after stability +2024-07-05T13:35:55.247Z [INFO] server-e10b353c-f776-a492-b0ac-a723a30b6647: starting snapshot up to: index=12 +2024-07-05T13:35:55.247Z [INFO] creating new snapshot: path=/tmp/raft980108311/snapshots/2-12-1720186555247.tmp +2024-07-05T13:35:55.255Z [INFO] server-e10b353c-f776-a492-b0ac-a723a30b6647: snapshot complete up to: index=12 +2024-07-05T13:35:55.270Z [INFO] server-f76801ba-f351-af83-30e1-172c01eac430: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f76801ba-f351-af83-30e1-172c01eac430 Address:f76801ba-f351-af83-30e1-172c01eac430} {Suffrage:Voter ID:server-58d81628-5a40-4c5f-e008-eec09104103f Address:58d81628-5a40-4c5f-e008-eec09104103f} {Suffrage:Voter ID:server-815cf460-f432-12bd-a2d0-77cd0af17388 Address:815cf460-f432-12bd-a2d0-77cd0af17388}]" +2024-07-05T13:35:55.270Z [INFO] server-f76801ba-f351-af83-30e1-172c01eac430: entering follower state: follower="Node at f76801ba-f351-af83-30e1-172c01eac430 [Follower]" leader-address= leader-id= +2024-07-05T13:35:55.271Z [INFO] server-58d81628-5a40-4c5f-e008-eec09104103f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f76801ba-f351-af83-30e1-172c01eac430 Address:f76801ba-f351-af83-30e1-172c01eac430} {Suffrage:Voter ID:server-58d81628-5a40-4c5f-e008-eec09104103f Address:58d81628-5a40-4c5f-e008-eec09104103f} {Suffrage:Voter ID:server-815cf460-f432-12bd-a2d0-77cd0af17388 Address:815cf460-f432-12bd-a2d0-77cd0af17388}]" +2024-07-05T13:35:55.271Z [INFO] server-58d81628-5a40-4c5f-e008-eec09104103f: entering follower state: follower="Node at 58d81628-5a40-4c5f-e008-eec09104103f [Follower]" leader-address= leader-id= +2024-07-05T13:35:55.272Z [INFO] server-815cf460-f432-12bd-a2d0-77cd0af17388: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f76801ba-f351-af83-30e1-172c01eac430 Address:f76801ba-f351-af83-30e1-172c01eac430} {Suffrage:Voter ID:server-58d81628-5a40-4c5f-e008-eec09104103f Address:58d81628-5a40-4c5f-e008-eec09104103f} {Suffrage:Voter ID:server-815cf460-f432-12bd-a2d0-77cd0af17388 Address:815cf460-f432-12bd-a2d0-77cd0af17388}]" +2024-07-05T13:35:55.272Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:55.272Z [INFO] server-815cf460-f432-12bd-a2d0-77cd0af17388: entering follower state: follower="Node at 815cf460-f432-12bd-a2d0-77cd0af17388 [Follower]" leader-address= leader-id= +2024-07-05T13:35:56.001Z [WARN] server-815cf460-f432-12bd-a2d0-77cd0af17388: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:56.001Z [INFO] server-815cf460-f432-12bd-a2d0-77cd0af17388: entering candidate state: node="Node at 815cf460-f432-12bd-a2d0-77cd0af17388 [Candidate]" term=2 +2024-07-05T13:35:56.001Z [INFO] server-815cf460-f432-12bd-a2d0-77cd0af17388: election won: term=2 tally=2 +2024-07-05T13:35:56.002Z [INFO] server-815cf460-f432-12bd-a2d0-77cd0af17388: entering leader state: leader="Node at 815cf460-f432-12bd-a2d0-77cd0af17388 [Leader]" +2024-07-05T13:35:56.002Z [INFO] server-815cf460-f432-12bd-a2d0-77cd0af17388: added peer, starting replication: peer=server-f76801ba-f351-af83-30e1-172c01eac430 +2024-07-05T13:35:56.002Z [INFO] server-815cf460-f432-12bd-a2d0-77cd0af17388: added peer, starting replication: peer=server-58d81628-5a40-4c5f-e008-eec09104103f +2024-07-05T13:35:56.003Z [INFO] server-815cf460-f432-12bd-a2d0-77cd0af17388: pipelining replication: peer="{Voter server-f76801ba-f351-af83-30e1-172c01eac430 f76801ba-f351-af83-30e1-172c01eac430}" +2024-07-05T13:35:56.003Z [INFO] server-815cf460-f432-12bd-a2d0-77cd0af17388: pipelining replication: peer="{Voter server-58d81628-5a40-4c5f-e008-eec09104103f 58d81628-5a40-4c5f-e008-eec09104103f}" +2024-07-05T13:35:57.007Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:56.002064258 +0000 UTC m=+21.997985710 (1 nodes), 729.577423ms from start of poll, 732.713074ms from cluster start. Timeout at 2024-07-05 13:35:57.007092776 +0000 UTC m=+23.003013936, 1.005028226s after stability +2024-07-05T13:35:57.008Z [INFO] creating new snapshot: path=/tmp/raft3771443483/snapshots/2-13-1720186557008.tmp +2024-07-05T13:35:57.012Z [INFO] server-815cf460-f432-12bd-a2d0-77cd0af17388: copied to local snapshot: bytes=71 +2024-07-05T13:35:57.013Z [INFO] server-815cf460-f432-12bd-a2d0-77cd0af17388: snapshot restore progress: id=2-13-1720186557008 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:35:57.014Z [INFO] server-815cf460-f432-12bd-a2d0-77cd0af17388: restored user snapshot: index=13 +2024-07-05T13:35:57.014Z [ERROR] server-815cf460-f432-12bd-a2d0-77cd0af17388: failed to get log: index=3 error="log not found" +2024-07-05T13:35:57.014Z [ERROR] server-815cf460-f432-12bd-a2d0-77cd0af17388: failed to get log: index=3 error="log not found" +2024-07-05T13:35:57.014Z [INFO] server-815cf460-f432-12bd-a2d0-77cd0af17388: aborting pipeline replication: peer="{Voter server-58d81628-5a40-4c5f-e008-eec09104103f 58d81628-5a40-4c5f-e008-eec09104103f}" +2024-07-05T13:35:57.014Z [INFO] server-815cf460-f432-12bd-a2d0-77cd0af17388: aborting pipeline replication: peer="{Voter server-f76801ba-f351-af83-30e1-172c01eac430 f76801ba-f351-af83-30e1-172c01eac430}" +2024-07-05T13:35:57.020Z [ERROR] server-815cf460-f432-12bd-a2d0-77cd0af17388: failed to get log: index=3 error="log not found" +2024-07-05T13:35:57.023Z [INFO] creating new snapshot: path=/tmp/raft1669446978/snapshots/2-13-1720186557023.tmp +2024-07-05T13:35:57.024Z [ERROR] server-815cf460-f432-12bd-a2d0-77cd0af17388: failed to get log: index=3 error="log not found" +2024-07-05T13:35:57.025Z [INFO] server-58d81628-5a40-4c5f-e008-eec09104103f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:35:57.026Z [INFO] creating new snapshot: path=/tmp/raft1636084885/snapshots/2-13-1720186557026.tmp +2024-07-05T13:35:57.028Z [INFO] server-58d81628-5a40-4c5f-e008-eec09104103f: copied to local snapshot: bytes=71 +2024-07-05T13:35:57.028Z [INFO] server-f76801ba-f351-af83-30e1-172c01eac430: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:35:57.030Z [INFO] server-58d81628-5a40-4c5f-e008-eec09104103f: snapshot restore progress: id=2-13-1720186557023 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:35:57.030Z [INFO] server-f76801ba-f351-af83-30e1-172c01eac430: copied to local snapshot: bytes=71 +2024-07-05T13:35:57.030Z [INFO] server-58d81628-5a40-4c5f-e008-eec09104103f: Installed remote snapshot +2024-07-05T13:35:57.031Z [INFO] server-815cf460-f432-12bd-a2d0-77cd0af17388: pipelining replication: peer="{Voter server-58d81628-5a40-4c5f-e008-eec09104103f 58d81628-5a40-4c5f-e008-eec09104103f}" +2024-07-05T13:35:57.031Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:35:57.032Z [INFO] server-f76801ba-f351-af83-30e1-172c01eac430: snapshot restore progress: id=2-13-1720186557026 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:35:57.032Z [INFO] server-f76801ba-f351-af83-30e1-172c01eac430: Installed remote snapshot +2024-07-05T13:35:57.033Z [INFO] server-815cf460-f432-12bd-a2d0-77cd0af17388: pipelining replication: peer="{Voter server-f76801ba-f351-af83-30e1-172c01eac430 f76801ba-f351-af83-30e1-172c01eac430}" +2024-07-05T13:35:58.038Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:35:57.032135402 +0000 UTC m=+23.028056853 (2 nodes), 9.625µs from start of poll, 1.762784217s from cluster start. Timeout at 2024-07-05 13:35:58.038022278 +0000 UTC m=+24.033943730, 1.005886877s after stability +2024-07-05T13:35:58.055Z [INFO] server-e10b353c-f776-a492-b0ac-a723a30b6647: aborting pipeline replication: peer="{Voter server-e0719c68-24ef-5287-6063-394c93632913 e0719c68-24ef-5287-6063-394c93632913}" +2024-07-05T13:35:58.055Z [INFO] server-e10b353c-f776-a492-b0ac-a723a30b6647: aborting pipeline replication: peer="{Voter server-b826c733-e625-48cd-a25a-c753aab03365 b826c733-e625-48cd-a25a-c753aab03365}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true -2024-07-05T13:29:42.814Z [INFO] server-673cb62d-279d-5045-65f2-fe38d54b601f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-673cb62d-279d-5045-65f2-fe38d54b601f Address:673cb62d-279d-5045-65f2-fe38d54b601f} {Suffrage:Voter ID:server-d914dc04-2cfe-74a6-9ffc-3e43fee9a698 Address:d914dc04-2cfe-74a6-9ffc-3e43fee9a698} {Suffrage:Voter ID:server-8cf89a20-88e8-0e92-a9b0-f4217d8ee9e6 Address:8cf89a20-88e8-0e92-a9b0-f4217d8ee9e6}]" -2024-07-05T13:29:42.814Z [INFO] server-d914dc04-2cfe-74a6-9ffc-3e43fee9a698: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-673cb62d-279d-5045-65f2-fe38d54b601f Address:673cb62d-279d-5045-65f2-fe38d54b601f} {Suffrage:Voter ID:server-d914dc04-2cfe-74a6-9ffc-3e43fee9a698 Address:d914dc04-2cfe-74a6-9ffc-3e43fee9a698} {Suffrage:Voter ID:server-8cf89a20-88e8-0e92-a9b0-f4217d8ee9e6 Address:8cf89a20-88e8-0e92-a9b0-f4217d8ee9e6}]" -2024-07-05T13:29:42.815Z [INFO] server-8cf89a20-88e8-0e92-a9b0-f4217d8ee9e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-673cb62d-279d-5045-65f2-fe38d54b601f Address:673cb62d-279d-5045-65f2-fe38d54b601f} {Suffrage:Voter ID:server-d914dc04-2cfe-74a6-9ffc-3e43fee9a698 Address:d914dc04-2cfe-74a6-9ffc-3e43fee9a698} {Suffrage:Voter ID:server-8cf89a20-88e8-0e92-a9b0-f4217d8ee9e6 Address:8cf89a20-88e8-0e92-a9b0-f4217d8ee9e6}]" -2024-07-05T13:29:42.815Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:42.815Z [INFO] server-673cb62d-279d-5045-65f2-fe38d54b601f: entering follower state: follower="Node at 673cb62d-279d-5045-65f2-fe38d54b601f [Follower]" leader-address= leader-id= -2024-07-05T13:29:42.815Z [INFO] server-d914dc04-2cfe-74a6-9ffc-3e43fee9a698: entering follower state: follower="Node at d914dc04-2cfe-74a6-9ffc-3e43fee9a698 [Follower]" leader-address= leader-id= -2024-07-05T13:29:42.815Z [INFO] server-8cf89a20-88e8-0e92-a9b0-f4217d8ee9e6: entering follower state: follower="Node at 8cf89a20-88e8-0e92-a9b0-f4217d8ee9e6 [Follower]" leader-address= leader-id= -2024-07-05T13:29:43.448Z [WARN] server-d914dc04-2cfe-74a6-9ffc-3e43fee9a698: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:43.448Z [INFO] server-d914dc04-2cfe-74a6-9ffc-3e43fee9a698: entering candidate state: node="Node at d914dc04-2cfe-74a6-9ffc-3e43fee9a698 [Candidate]" term=2 -2024-07-05T13:29:43.448Z [INFO] server-d914dc04-2cfe-74a6-9ffc-3e43fee9a698: election won: term=2 tally=2 -2024-07-05T13:29:43.448Z [INFO] server-d914dc04-2cfe-74a6-9ffc-3e43fee9a698: entering leader state: leader="Node at d914dc04-2cfe-74a6-9ffc-3e43fee9a698 [Leader]" -2024-07-05T13:29:43.448Z [INFO] server-d914dc04-2cfe-74a6-9ffc-3e43fee9a698: added peer, starting replication: peer=server-673cb62d-279d-5045-65f2-fe38d54b601f -2024-07-05T13:29:43.448Z [INFO] server-d914dc04-2cfe-74a6-9ffc-3e43fee9a698: added peer, starting replication: peer=server-8cf89a20-88e8-0e92-a9b0-f4217d8ee9e6 -2024-07-05T13:29:43.448Z [INFO] server-d914dc04-2cfe-74a6-9ffc-3e43fee9a698: pipelining replication: peer="{Voter server-8cf89a20-88e8-0e92-a9b0-f4217d8ee9e6 8cf89a20-88e8-0e92-a9b0-f4217d8ee9e6}" -2024-07-05T13:29:43.449Z [INFO] server-d914dc04-2cfe-74a6-9ffc-3e43fee9a698: pipelining replication: peer="{Voter server-673cb62d-279d-5045-65f2-fe38d54b601f 673cb62d-279d-5045-65f2-fe38d54b601f}" -2024-07-05T13:29:44.455Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:43.449200142 +0000 UTC m=+24.219110817 (1 nodes), 634.022158ms from start of poll, 634.860568ms from cluster start. Timeout at 2024-07-05 13:29:44.45576995 +0000 UTC m=+25.225680625, 1.006569808s after stability -2024-07-05T13:29:44.457Z [INFO] server-d914dc04-2cfe-74a6-9ffc-3e43fee9a698: starting snapshot up to: index=12 -2024-07-05T13:29:44.457Z [INFO] creating new snapshot: path=/tmp/raft348635024/snapshots/2-12-1720186184457.tmp -2024-07-05T13:29:44.459Z [INFO] server-d914dc04-2cfe-74a6-9ffc-3e43fee9a698: snapshot complete up to: index=12 -2024-07-05T13:29:44.462Z [INFO] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf Address:c9ce5d77-bfb1-1652-7a1c-f50ae12460cf} {Suffrage:Voter ID:server-e8853c3a-efaa-4139-5d37-766db7a14205 Address:e8853c3a-efaa-4139-5d37-766db7a14205} {Suffrage:Voter ID:server-175645fa-b3f3-1c45-1539-677698d7db37 Address:175645fa-b3f3-1c45-1539-677698d7db37}]" -2024-07-05T13:29:44.462Z [INFO] server-e8853c3a-efaa-4139-5d37-766db7a14205: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf Address:c9ce5d77-bfb1-1652-7a1c-f50ae12460cf} {Suffrage:Voter ID:server-e8853c3a-efaa-4139-5d37-766db7a14205 Address:e8853c3a-efaa-4139-5d37-766db7a14205} {Suffrage:Voter ID:server-175645fa-b3f3-1c45-1539-677698d7db37 Address:175645fa-b3f3-1c45-1539-677698d7db37}]" -2024-07-05T13:29:44.463Z [INFO] server-175645fa-b3f3-1c45-1539-677698d7db37: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf Address:c9ce5d77-bfb1-1652-7a1c-f50ae12460cf} {Suffrage:Voter ID:server-e8853c3a-efaa-4139-5d37-766db7a14205 Address:e8853c3a-efaa-4139-5d37-766db7a14205} {Suffrage:Voter ID:server-175645fa-b3f3-1c45-1539-677698d7db37 Address:175645fa-b3f3-1c45-1539-677698d7db37}]" -2024-07-05T13:29:44.463Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:44.463Z [INFO] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: entering follower state: follower="Node at c9ce5d77-bfb1-1652-7a1c-f50ae12460cf [Follower]" leader-address= leader-id= -2024-07-05T13:29:44.463Z [INFO] server-e8853c3a-efaa-4139-5d37-766db7a14205: entering follower state: follower="Node at e8853c3a-efaa-4139-5d37-766db7a14205 [Follower]" leader-address= leader-id= -2024-07-05T13:29:44.463Z [INFO] server-175645fa-b3f3-1c45-1539-677698d7db37: entering follower state: follower="Node at 175645fa-b3f3-1c45-1539-677698d7db37 [Follower]" leader-address= leader-id= -2024-07-05T13:29:44.972Z [WARN] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:44.972Z [INFO] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: entering candidate state: node="Node at c9ce5d77-bfb1-1652-7a1c-f50ae12460cf [Candidate]" term=2 -2024-07-05T13:29:44.972Z [INFO] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: election won: term=2 tally=2 -2024-07-05T13:29:44.972Z [INFO] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: entering leader state: leader="Node at c9ce5d77-bfb1-1652-7a1c-f50ae12460cf [Leader]" -2024-07-05T13:29:44.972Z [INFO] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: added peer, starting replication: peer=server-e8853c3a-efaa-4139-5d37-766db7a14205 -2024-07-05T13:29:44.972Z [INFO] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: added peer, starting replication: peer=server-175645fa-b3f3-1c45-1539-677698d7db37 -2024-07-05T13:29:44.972Z [INFO] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: pipelining replication: peer="{Voter server-175645fa-b3f3-1c45-1539-677698d7db37 175645fa-b3f3-1c45-1539-677698d7db37}" -2024-07-05T13:29:44.973Z [INFO] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: pipelining replication: peer="{Voter server-e8853c3a-efaa-4139-5d37-766db7a14205 e8853c3a-efaa-4139-5d37-766db7a14205}" -2024-07-05T13:29:45.979Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:44.973181159 +0000 UTC m=+25.743091834 (1 nodes), 509.95943ms from start of poll, 510.697644ms from cluster start. Timeout at 2024-07-05 13:29:45.979072591 +0000 UTC m=+26.748983266, 1.005891432s after stability -2024-07-05T13:29:45.979Z [INFO] creating new snapshot: path=/tmp/raft3359360205/snapshots/2-13-1720186185979.tmp -2024-07-05T13:29:45.981Z [INFO] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: copied to local snapshot: bytes=71 -2024-07-05T13:29:45.981Z [INFO] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: snapshot restore progress: id=2-13-1720186185979 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:45.981Z [INFO] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: removing all old logs from log store -2024-07-05T13:29:45.981Z [INFO] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: compacting logs: from=1 to=2 -2024-07-05T13:29:45.981Z [INFO] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: restored user snapshot: index=13 -2024-07-05T13:29:45.982Z [ERROR] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: failed to get log: index=2 error="log not found" -2024-07-05T13:29:45.982Z [INFO] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: aborting pipeline replication: peer="{Voter server-e8853c3a-efaa-4139-5d37-766db7a14205 e8853c3a-efaa-4139-5d37-766db7a14205}" -2024-07-05T13:29:45.982Z [ERROR] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: failed to get log: index=2 error="log not found" -2024-07-05T13:29:45.982Z [INFO] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: aborting pipeline replication: peer="{Voter server-175645fa-b3f3-1c45-1539-677698d7db37 175645fa-b3f3-1c45-1539-677698d7db37}" -2024-07-05T13:29:45.989Z [ERROR] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: failed to get log: index=2 error="log not found" -2024-07-05T13:29:45.990Z [INFO] creating new snapshot: path=/tmp/raft4197369547/snapshots/2-13-1720186185990.tmp -2024-07-05T13:29:45.990Z [INFO] server-e8853c3a-efaa-4139-5d37-766db7a14205: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:45.991Z [INFO] server-e8853c3a-efaa-4139-5d37-766db7a14205: copied to local snapshot: bytes=71 -2024-07-05T13:29:45.991Z [ERROR] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: failed to get log: index=2 error="log not found" -2024-07-05T13:29:45.992Z [INFO] creating new snapshot: path=/tmp/raft2774151991/snapshots/2-13-1720186185992.tmp -2024-07-05T13:29:45.993Z [INFO] server-175645fa-b3f3-1c45-1539-677698d7db37: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:45.993Z [INFO] server-175645fa-b3f3-1c45-1539-677698d7db37: copied to local snapshot: bytes=71 -2024-07-05T13:29:45.994Z [INFO] server-e8853c3a-efaa-4139-5d37-766db7a14205: snapshot restore progress: id=2-13-1720186185990 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:45.994Z [INFO] server-e8853c3a-efaa-4139-5d37-766db7a14205: removing all old logs from log store -2024-07-05T13:29:45.994Z [INFO] server-e8853c3a-efaa-4139-5d37-766db7a14205: compacting logs: from=1 to=2 -2024-07-05T13:29:45.994Z [INFO] server-e8853c3a-efaa-4139-5d37-766db7a14205: Installed remote snapshot -2024-07-05T13:29:45.999Z [INFO] server-175645fa-b3f3-1c45-1539-677698d7db37: snapshot restore progress: id=2-13-1720186185992 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:45.999Z [INFO] server-175645fa-b3f3-1c45-1539-677698d7db37: removing all old logs from log store -2024-07-05T13:29:45.999Z [INFO] server-175645fa-b3f3-1c45-1539-677698d7db37: compacting logs: from=1 to=2 -2024-07-05T13:29:45.999Z [INFO] server-175645fa-b3f3-1c45-1539-677698d7db37: Installed remote snapshot -2024-07-05T13:29:45.999Z [INFO] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: pipelining replication: peer="{Voter server-175645fa-b3f3-1c45-1539-677698d7db37 175645fa-b3f3-1c45-1539-677698d7db37}" -2024-07-05T13:29:45.999Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:29:46.000Z [INFO] server-c9ce5d77-bfb1-1652-7a1c-f50ae12460cf: pipelining replication: peer="{Voter server-e8853c3a-efaa-4139-5d37-766db7a14205 e8853c3a-efaa-4139-5d37-766db7a14205}" -2024-07-05T13:29:47.004Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:29:45.999576671 +0000 UTC m=+26.769487346 (2 nodes), 2.12µs from start of poll, 1.537093156s from cluster start. Timeout at 2024-07-05 13:29:47.004848565 +0000 UTC m=+27.774759240, 1.005271894s after stability -2024-07-05T13:29:47.018Z [INFO] server-d914dc04-2cfe-74a6-9ffc-3e43fee9a698: aborting pipeline replication: peer="{Voter server-8cf89a20-88e8-0e92-a9b0-f4217d8ee9e6 8cf89a20-88e8-0e92-a9b0-f4217d8ee9e6}" -2024-07-05T13:29:47.018Z [INFO] server-d914dc04-2cfe-74a6-9ffc-3e43fee9a698: aborting pipeline replication: peer="{Voter server-673cb62d-279d-5045-65f2-fe38d54b601f 673cb62d-279d-5045-65f2-fe38d54b601f}" +2024-07-05T13:35:58.067Z [INFO] server-4f4b299c-f060-79ba-41e2-137be2881963: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f4b299c-f060-79ba-41e2-137be2881963 Address:4f4b299c-f060-79ba-41e2-137be2881963} {Suffrage:Voter ID:server-25d96bca-9655-5919-588e-e65bfabfd2aa Address:25d96bca-9655-5919-588e-e65bfabfd2aa} {Suffrage:Voter ID:server-55d9398d-58c3-07c5-7f5d-8b62a644833c Address:55d9398d-58c3-07c5-7f5d-8b62a644833c}]" +2024-07-05T13:35:58.067Z [INFO] server-4f4b299c-f060-79ba-41e2-137be2881963: entering follower state: follower="Node at 4f4b299c-f060-79ba-41e2-137be2881963 [Follower]" leader-address= leader-id= +2024-07-05T13:35:58.067Z [INFO] server-25d96bca-9655-5919-588e-e65bfabfd2aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f4b299c-f060-79ba-41e2-137be2881963 Address:4f4b299c-f060-79ba-41e2-137be2881963} {Suffrage:Voter ID:server-25d96bca-9655-5919-588e-e65bfabfd2aa Address:25d96bca-9655-5919-588e-e65bfabfd2aa} {Suffrage:Voter ID:server-55d9398d-58c3-07c5-7f5d-8b62a644833c Address:55d9398d-58c3-07c5-7f5d-8b62a644833c}]" +2024-07-05T13:35:58.068Z [INFO] server-25d96bca-9655-5919-588e-e65bfabfd2aa: entering follower state: follower="Node at 25d96bca-9655-5919-588e-e65bfabfd2aa [Follower]" leader-address= leader-id= +2024-07-05T13:35:58.068Z [INFO] server-55d9398d-58c3-07c5-7f5d-8b62a644833c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f4b299c-f060-79ba-41e2-137be2881963 Address:4f4b299c-f060-79ba-41e2-137be2881963} {Suffrage:Voter ID:server-25d96bca-9655-5919-588e-e65bfabfd2aa Address:25d96bca-9655-5919-588e-e65bfabfd2aa} {Suffrage:Voter ID:server-55d9398d-58c3-07c5-7f5d-8b62a644833c Address:55d9398d-58c3-07c5-7f5d-8b62a644833c}]" +2024-07-05T13:35:58.069Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:58.069Z [INFO] server-55d9398d-58c3-07c5-7f5d-8b62a644833c: entering follower state: follower="Node at 55d9398d-58c3-07c5-7f5d-8b62a644833c [Follower]" leader-address= leader-id= +2024-07-05T13:35:58.680Z [WARN] server-55d9398d-58c3-07c5-7f5d-8b62a644833c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:35:58.682Z [INFO] server-55d9398d-58c3-07c5-7f5d-8b62a644833c: entering candidate state: node="Node at 55d9398d-58c3-07c5-7f5d-8b62a644833c [Candidate]" term=2 +2024-07-05T13:35:58.684Z [INFO] server-55d9398d-58c3-07c5-7f5d-8b62a644833c: election won: term=2 tally=2 +2024-07-05T13:35:58.684Z [INFO] server-55d9398d-58c3-07c5-7f5d-8b62a644833c: entering leader state: leader="Node at 55d9398d-58c3-07c5-7f5d-8b62a644833c [Leader]" +2024-07-05T13:35:58.684Z [INFO] server-55d9398d-58c3-07c5-7f5d-8b62a644833c: added peer, starting replication: peer=server-4f4b299c-f060-79ba-41e2-137be2881963 +2024-07-05T13:35:58.684Z [INFO] server-55d9398d-58c3-07c5-7f5d-8b62a644833c: added peer, starting replication: peer=server-25d96bca-9655-5919-588e-e65bfabfd2aa +2024-07-05T13:35:58.687Z [INFO] server-55d9398d-58c3-07c5-7f5d-8b62a644833c: pipelining replication: peer="{Voter server-4f4b299c-f060-79ba-41e2-137be2881963 4f4b299c-f060-79ba-41e2-137be2881963}" +2024-07-05T13:35:58.687Z [INFO] server-55d9398d-58c3-07c5-7f5d-8b62a644833c: pipelining replication: peer="{Voter server-25d96bca-9655-5919-588e-e65bfabfd2aa 25d96bca-9655-5919-588e-e65bfabfd2aa}" +2024-07-05T13:35:59.692Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:35:58.686810386 +0000 UTC m=+24.682731837 (1 nodes), 617.533095ms from start of poll, 620.447083ms from cluster start. Timeout at 2024-07-05 13:35:59.692513516 +0000 UTC m=+25.688434967, 1.00570313s after stability +2024-07-05T13:35:59.695Z [INFO] server-55d9398d-58c3-07c5-7f5d-8b62a644833c: starting snapshot up to: index=12 +2024-07-05T13:35:59.695Z [INFO] creating new snapshot: path=/tmp/raft962523949/snapshots/2-12-1720186559695.tmp +2024-07-05T13:35:59.696Z [INFO] server-55d9398d-58c3-07c5-7f5d-8b62a644833c: snapshot complete up to: index=12 +2024-07-05T13:35:59.699Z [INFO] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8a671c2-ea05-66b4-f357-8ac7c25b0306 Address:d8a671c2-ea05-66b4-f357-8ac7c25b0306} {Suffrage:Voter ID:server-f15163a8-6b49-60f1-93d0-c27d023f838a Address:f15163a8-6b49-60f1-93d0-c27d023f838a} {Suffrage:Voter ID:server-c0c0a425-e135-ddff-a5e7-f89a53f4e3c0 Address:c0c0a425-e135-ddff-a5e7-f89a53f4e3c0}]" +2024-07-05T13:35:59.700Z [INFO] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: entering follower state: follower="Node at d8a671c2-ea05-66b4-f357-8ac7c25b0306 [Follower]" leader-address= leader-id= +2024-07-05T13:35:59.700Z [INFO] server-f15163a8-6b49-60f1-93d0-c27d023f838a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8a671c2-ea05-66b4-f357-8ac7c25b0306 Address:d8a671c2-ea05-66b4-f357-8ac7c25b0306} {Suffrage:Voter ID:server-f15163a8-6b49-60f1-93d0-c27d023f838a Address:f15163a8-6b49-60f1-93d0-c27d023f838a} {Suffrage:Voter ID:server-c0c0a425-e135-ddff-a5e7-f89a53f4e3c0 Address:c0c0a425-e135-ddff-a5e7-f89a53f4e3c0}]" +2024-07-05T13:35:59.700Z [INFO] server-f15163a8-6b49-60f1-93d0-c27d023f838a: entering follower state: follower="Node at f15163a8-6b49-60f1-93d0-c27d023f838a [Follower]" leader-address= leader-id= +2024-07-05T13:35:59.700Z [INFO] server-c0c0a425-e135-ddff-a5e7-f89a53f4e3c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8a671c2-ea05-66b4-f357-8ac7c25b0306 Address:d8a671c2-ea05-66b4-f357-8ac7c25b0306} {Suffrage:Voter ID:server-f15163a8-6b49-60f1-93d0-c27d023f838a Address:f15163a8-6b49-60f1-93d0-c27d023f838a} {Suffrage:Voter ID:server-c0c0a425-e135-ddff-a5e7-f89a53f4e3c0 Address:c0c0a425-e135-ddff-a5e7-f89a53f4e3c0}]" +2024-07-05T13:35:59.700Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:35:59.700Z [INFO] server-c0c0a425-e135-ddff-a5e7-f89a53f4e3c0: entering follower state: follower="Node at c0c0a425-e135-ddff-a5e7-f89a53f4e3c0 [Follower]" leader-address= leader-id= +2024-07-05T13:36:00.649Z [WARN] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:00.649Z [INFO] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: entering candidate state: node="Node at d8a671c2-ea05-66b4-f357-8ac7c25b0306 [Candidate]" term=2 +2024-07-05T13:36:00.650Z [INFO] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: election won: term=2 tally=2 +2024-07-05T13:36:00.650Z [INFO] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: entering leader state: leader="Node at d8a671c2-ea05-66b4-f357-8ac7c25b0306 [Leader]" +2024-07-05T13:36:00.650Z [INFO] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: added peer, starting replication: peer=server-f15163a8-6b49-60f1-93d0-c27d023f838a +2024-07-05T13:36:00.650Z [INFO] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: added peer, starting replication: peer=server-c0c0a425-e135-ddff-a5e7-f89a53f4e3c0 +2024-07-05T13:36:00.651Z [INFO] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: pipelining replication: peer="{Voter server-f15163a8-6b49-60f1-93d0-c27d023f838a f15163a8-6b49-60f1-93d0-c27d023f838a}" +2024-07-05T13:36:00.651Z [INFO] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: pipelining replication: peer="{Voter server-c0c0a425-e135-ddff-a5e7-f89a53f4e3c0 c0c0a425-e135-ddff-a5e7-f89a53f4e3c0}" +2024-07-05T13:36:01.657Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:00.650345978 +0000 UTC m=+26.646267430 (1 nodes), 949.772985ms from start of poll, 950.731676ms from cluster start. Timeout at 2024-07-05 13:36:01.656866344 +0000 UTC m=+27.652787795, 1.006520365s after stability +2024-07-05T13:36:01.658Z [INFO] creating new snapshot: path=/tmp/raft1207881658/snapshots/2-13-1720186561658.tmp +2024-07-05T13:36:01.662Z [INFO] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: copied to local snapshot: bytes=71 +2024-07-05T13:36:01.663Z [INFO] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: snapshot restore progress: id=2-13-1720186561658 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:01.664Z [INFO] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: removing all old logs from log store +2024-07-05T13:36:01.664Z [INFO] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: compacting logs: from=1 to=2 +2024-07-05T13:36:01.664Z [INFO] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: restored user snapshot: index=13 +2024-07-05T13:36:01.664Z [ERROR] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: failed to get log: index=2 error="log not found" +2024-07-05T13:36:01.664Z [INFO] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: aborting pipeline replication: peer="{Voter server-c0c0a425-e135-ddff-a5e7-f89a53f4e3c0 c0c0a425-e135-ddff-a5e7-f89a53f4e3c0}" +2024-07-05T13:36:01.664Z [ERROR] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: failed to get log: index=2 error="log not found" +2024-07-05T13:36:01.665Z [INFO] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: aborting pipeline replication: peer="{Voter server-f15163a8-6b49-60f1-93d0-c27d023f838a f15163a8-6b49-60f1-93d0-c27d023f838a}" +2024-07-05T13:36:01.670Z [ERROR] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: failed to get log: index=2 error="log not found" +2024-07-05T13:36:01.672Z [INFO] creating new snapshot: path=/tmp/raft1316255635/snapshots/2-13-1720186561672.tmp +2024-07-05T13:36:01.680Z [ERROR] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: failed to get log: index=2 error="log not found" +2024-07-05T13:36:01.682Z [INFO] server-f15163a8-6b49-60f1-93d0-c27d023f838a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:01.684Z [INFO] creating new snapshot: path=/tmp/raft2409056178/snapshots/2-13-1720186561683.tmp +2024-07-05T13:36:01.685Z [INFO] server-f15163a8-6b49-60f1-93d0-c27d023f838a: copied to local snapshot: bytes=71 +2024-07-05T13:36:01.686Z [INFO] server-c0c0a425-e135-ddff-a5e7-f89a53f4e3c0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:01.686Z [INFO] server-f15163a8-6b49-60f1-93d0-c27d023f838a: snapshot restore progress: id=2-13-1720186561672 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:01.686Z [INFO] server-f15163a8-6b49-60f1-93d0-c27d023f838a: removing all old logs from log store +2024-07-05T13:36:01.686Z [INFO] server-f15163a8-6b49-60f1-93d0-c27d023f838a: compacting logs: from=1 to=2 +2024-07-05T13:36:01.687Z [INFO] server-f15163a8-6b49-60f1-93d0-c27d023f838a: Installed remote snapshot +2024-07-05T13:36:01.688Z [INFO] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: pipelining replication: peer="{Voter server-f15163a8-6b49-60f1-93d0-c27d023f838a f15163a8-6b49-60f1-93d0-c27d023f838a}" +2024-07-05T13:36:01.688Z [INFO] server-c0c0a425-e135-ddff-a5e7-f89a53f4e3c0: copied to local snapshot: bytes=71 +2024-07-05T13:36:01.688Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:36:01.689Z [INFO] server-c0c0a425-e135-ddff-a5e7-f89a53f4e3c0: snapshot restore progress: id=2-13-1720186561683 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:01.690Z [INFO] server-c0c0a425-e135-ddff-a5e7-f89a53f4e3c0: removing all old logs from log store +2024-07-05T13:36:01.690Z [INFO] server-c0c0a425-e135-ddff-a5e7-f89a53f4e3c0: compacting logs: from=1 to=2 +2024-07-05T13:36:01.690Z [INFO] server-c0c0a425-e135-ddff-a5e7-f89a53f4e3c0: Installed remote snapshot +2024-07-05T13:36:01.691Z [INFO] server-d8a671c2-ea05-66b4-f357-8ac7c25b0306: pipelining replication: peer="{Voter server-c0c0a425-e135-ddff-a5e7-f89a53f4e3c0 c0c0a425-e135-ddff-a5e7-f89a53f4e3c0}" +2024-07-05T13:36:02.694Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:36:01.688781677 +0000 UTC m=+27.684702837 (2 nodes), 9.041µs from start of poll, 1.989167083s from cluster start. Timeout at 2024-07-05 13:36:02.693933276 +0000 UTC m=+28.689855019, 1.005152182s after stability +2024-07-05T13:36:02.709Z [INFO] server-55d9398d-58c3-07c5-7f5d-8b62a644833c: aborting pipeline replication: peer="{Voter server-25d96bca-9655-5919-588e-e65bfabfd2aa 25d96bca-9655-5919-588e-e65bfabfd2aa}" +2024-07-05T13:36:02.709Z [INFO] server-55d9398d-58c3-07c5-7f5d-8b62a644833c: aborting pipeline replication: peer="{Voter server-4f4b299c-f060-79ba-41e2-137be2881963 4f4b299c-f060-79ba-41e2-137be2881963}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false -2024-07-05T13:29:47.021Z [INFO] server-f3cc4401-8037-5df9-4aef-b334c7d9d58d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f3cc4401-8037-5df9-4aef-b334c7d9d58d Address:f3cc4401-8037-5df9-4aef-b334c7d9d58d} {Suffrage:Voter ID:server-51cd2995-3f44-70e7-d275-1210627193c7 Address:51cd2995-3f44-70e7-d275-1210627193c7} {Suffrage:Voter ID:server-c05b43fd-403a-81c3-f55f-a31d8bd834ad Address:c05b43fd-403a-81c3-f55f-a31d8bd834ad}]" -2024-07-05T13:29:47.021Z [INFO] server-51cd2995-3f44-70e7-d275-1210627193c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f3cc4401-8037-5df9-4aef-b334c7d9d58d Address:f3cc4401-8037-5df9-4aef-b334c7d9d58d} {Suffrage:Voter ID:server-51cd2995-3f44-70e7-d275-1210627193c7 Address:51cd2995-3f44-70e7-d275-1210627193c7} {Suffrage:Voter ID:server-c05b43fd-403a-81c3-f55f-a31d8bd834ad Address:c05b43fd-403a-81c3-f55f-a31d8bd834ad}]" -2024-07-05T13:29:47.022Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f3cc4401-8037-5df9-4aef-b334c7d9d58d Address:f3cc4401-8037-5df9-4aef-b334c7d9d58d} {Suffrage:Voter ID:server-51cd2995-3f44-70e7-d275-1210627193c7 Address:51cd2995-3f44-70e7-d275-1210627193c7} {Suffrage:Voter ID:server-c05b43fd-403a-81c3-f55f-a31d8bd834ad Address:c05b43fd-403a-81c3-f55f-a31d8bd834ad}]" -2024-07-05T13:29:47.022Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:47.022Z [INFO] server-f3cc4401-8037-5df9-4aef-b334c7d9d58d: entering follower state: follower="Node at f3cc4401-8037-5df9-4aef-b334c7d9d58d [Follower]" leader-address= leader-id= -2024-07-05T13:29:47.022Z [INFO] server-51cd2995-3f44-70e7-d275-1210627193c7: entering follower state: follower="Node at 51cd2995-3f44-70e7-d275-1210627193c7 [Follower]" leader-address= leader-id= -2024-07-05T13:29:47.022Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: entering follower state: follower="Node at c05b43fd-403a-81c3-f55f-a31d8bd834ad [Follower]" leader-address= leader-id= -2024-07-05T13:29:47.543Z [WARN] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:47.543Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: entering candidate state: node="Node at c05b43fd-403a-81c3-f55f-a31d8bd834ad [Candidate]" term=2 -2024-07-05T13:29:47.543Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: election won: term=2 tally=2 -2024-07-05T13:29:47.543Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: entering leader state: leader="Node at c05b43fd-403a-81c3-f55f-a31d8bd834ad [Leader]" -2024-07-05T13:29:47.543Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: added peer, starting replication: peer=server-f3cc4401-8037-5df9-4aef-b334c7d9d58d -2024-07-05T13:29:47.543Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: added peer, starting replication: peer=server-51cd2995-3f44-70e7-d275-1210627193c7 -2024-07-05T13:29:47.543Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: pipelining replication: peer="{Voter server-51cd2995-3f44-70e7-d275-1210627193c7 51cd2995-3f44-70e7-d275-1210627193c7}" -2024-07-05T13:29:47.543Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: pipelining replication: peer="{Voter server-f3cc4401-8037-5df9-4aef-b334c7d9d58d f3cc4401-8037-5df9-4aef-b334c7d9d58d}" -2024-07-05T13:29:48.549Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:47.543989331 +0000 UTC m=+28.313900006 (1 nodes), 521.822933ms from start of poll, 522.519749ms from cluster start. Timeout at 2024-07-05 13:29:48.54912619 +0000 UTC m=+29.319036865, 1.005136859s after stability -2024-07-05T13:29:48.556Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: starting snapshot up to: index=12 -2024-07-05T13:29:48.556Z [INFO] creating new snapshot: path=/tmp/raft1319562992/snapshots/2-12-1720186188556.tmp -2024-07-05T13:29:48.559Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: snapshot complete up to: index=12 -2024-07-05T13:29:48.564Z [INFO] creating new snapshot: path=/tmp/raft1319562992/snapshots/2-23-1720186188564.tmp -2024-07-05T13:29:48.565Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: copied to local snapshot: bytes=71 -2024-07-05T13:29:48.566Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: snapshot restore progress: id=2-23-1720186188564 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:48.566Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: restored user snapshot: index=23 -2024-07-05T13:29:48.566Z [ERROR] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: failed to get log: index=23 error="log not found" -2024-07-05T13:29:48.566Z [ERROR] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: failed to get log: index=23 error="log not found" -2024-07-05T13:29:48.566Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: aborting pipeline replication: peer="{Voter server-51cd2995-3f44-70e7-d275-1210627193c7 51cd2995-3f44-70e7-d275-1210627193c7}" -2024-07-05T13:29:48.566Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: aborting pipeline replication: peer="{Voter server-f3cc4401-8037-5df9-4aef-b334c7d9d58d f3cc4401-8037-5df9-4aef-b334c7d9d58d}" -2024-07-05T13:29:48.574Z [ERROR] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: failed to get log: index=23 error="log not found" -2024-07-05T13:29:48.574Z [INFO] creating new snapshot: path=/tmp/raft974459838/snapshots/2-23-1720186188574.tmp -2024-07-05T13:29:48.575Z [ERROR] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: failed to get log: index=23 error="log not found" -2024-07-05T13:29:48.576Z [INFO] creating new snapshot: path=/tmp/raft2760939530/snapshots/2-23-1720186188576.tmp -2024-07-05T13:29:48.577Z [INFO] server-f3cc4401-8037-5df9-4aef-b334c7d9d58d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:48.577Z [INFO] server-f3cc4401-8037-5df9-4aef-b334c7d9d58d: copied to local snapshot: bytes=71 -2024-07-05T13:29:48.579Z [INFO] server-51cd2995-3f44-70e7-d275-1210627193c7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:48.579Z [INFO] server-f3cc4401-8037-5df9-4aef-b334c7d9d58d: snapshot restore progress: id=2-23-1720186188576 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:48.579Z [INFO] server-f3cc4401-8037-5df9-4aef-b334c7d9d58d: Installed remote snapshot -2024-07-05T13:29:48.580Z [WARN] server-f3cc4401-8037-5df9-4aef-b334c7d9d58d: skipping application of old log: index=22 -2024-07-05T13:29:48.580Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: pipelining replication: peer="{Voter server-f3cc4401-8037-5df9-4aef-b334c7d9d58d f3cc4401-8037-5df9-4aef-b334c7d9d58d}" -2024-07-05T13:29:48.580Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:29:48.580Z [INFO] server-51cd2995-3f44-70e7-d275-1210627193c7: copied to local snapshot: bytes=71 -2024-07-05T13:29:48.581Z [INFO] server-51cd2995-3f44-70e7-d275-1210627193c7: snapshot restore progress: id=2-23-1720186188574 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:48.581Z [INFO] server-51cd2995-3f44-70e7-d275-1210627193c7: Installed remote snapshot -2024-07-05T13:29:48.581Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: pipelining replication: peer="{Voter server-51cd2995-3f44-70e7-d275-1210627193c7 51cd2995-3f44-70e7-d275-1210627193c7}" -2024-07-05T13:29:49.586Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:29:48.580391013 +0000 UTC m=+29.350301688 (2 nodes), 1.6µs from start of poll, 1.558921431s from cluster start. Timeout at 2024-07-05 13:29:49.586160649 +0000 UTC m=+30.356071364, 1.005769676s after stability -2024-07-05T13:29:49.598Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: aborting pipeline replication: peer="{Voter server-51cd2995-3f44-70e7-d275-1210627193c7 51cd2995-3f44-70e7-d275-1210627193c7}" -2024-07-05T13:29:49.598Z [INFO] server-c05b43fd-403a-81c3-f55f-a31d8bd834ad: aborting pipeline replication: peer="{Voter server-f3cc4401-8037-5df9-4aef-b334c7d9d58d f3cc4401-8037-5df9-4aef-b334c7d9d58d}" +2024-07-05T13:36:02.722Z [INFO] server-f14e4729-c511-7e3e-7fd0-5f12d8cc2692: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f14e4729-c511-7e3e-7fd0-5f12d8cc2692 Address:f14e4729-c511-7e3e-7fd0-5f12d8cc2692} {Suffrage:Voter ID:server-dd48df78-403e-0251-3804-d0d1e2c6e762 Address:dd48df78-403e-0251-3804-d0d1e2c6e762} {Suffrage:Voter ID:server-5536b621-794f-ff76-6e28-056070cb1441 Address:5536b621-794f-ff76-6e28-056070cb1441}]" +2024-07-05T13:36:02.722Z [INFO] server-f14e4729-c511-7e3e-7fd0-5f12d8cc2692: entering follower state: follower="Node at f14e4729-c511-7e3e-7fd0-5f12d8cc2692 [Follower]" leader-address= leader-id= +2024-07-05T13:36:02.723Z [INFO] server-dd48df78-403e-0251-3804-d0d1e2c6e762: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f14e4729-c511-7e3e-7fd0-5f12d8cc2692 Address:f14e4729-c511-7e3e-7fd0-5f12d8cc2692} {Suffrage:Voter ID:server-dd48df78-403e-0251-3804-d0d1e2c6e762 Address:dd48df78-403e-0251-3804-d0d1e2c6e762} {Suffrage:Voter ID:server-5536b621-794f-ff76-6e28-056070cb1441 Address:5536b621-794f-ff76-6e28-056070cb1441}]" +2024-07-05T13:36:02.723Z [INFO] server-dd48df78-403e-0251-3804-d0d1e2c6e762: entering follower state: follower="Node at dd48df78-403e-0251-3804-d0d1e2c6e762 [Follower]" leader-address= leader-id= +2024-07-05T13:36:02.723Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f14e4729-c511-7e3e-7fd0-5f12d8cc2692 Address:f14e4729-c511-7e3e-7fd0-5f12d8cc2692} {Suffrage:Voter ID:server-dd48df78-403e-0251-3804-d0d1e2c6e762 Address:dd48df78-403e-0251-3804-d0d1e2c6e762} {Suffrage:Voter ID:server-5536b621-794f-ff76-6e28-056070cb1441 Address:5536b621-794f-ff76-6e28-056070cb1441}]" +2024-07-05T13:36:02.724Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:02.724Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: entering follower state: follower="Node at 5536b621-794f-ff76-6e28-056070cb1441 [Follower]" leader-address= leader-id= +2024-07-05T13:36:03.508Z [WARN] server-5536b621-794f-ff76-6e28-056070cb1441: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:03.508Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: entering candidate state: node="Node at 5536b621-794f-ff76-6e28-056070cb1441 [Candidate]" term=2 +2024-07-05T13:36:03.509Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: election won: term=2 tally=2 +2024-07-05T13:36:03.509Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: entering leader state: leader="Node at 5536b621-794f-ff76-6e28-056070cb1441 [Leader]" +2024-07-05T13:36:03.509Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: added peer, starting replication: peer=server-f14e4729-c511-7e3e-7fd0-5f12d8cc2692 +2024-07-05T13:36:03.509Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: added peer, starting replication: peer=server-dd48df78-403e-0251-3804-d0d1e2c6e762 +2024-07-05T13:36:03.510Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: pipelining replication: peer="{Voter server-f14e4729-c511-7e3e-7fd0-5f12d8cc2692 f14e4729-c511-7e3e-7fd0-5f12d8cc2692}" +2024-07-05T13:36:03.512Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: pipelining replication: peer="{Voter server-dd48df78-403e-0251-3804-d0d1e2c6e762 dd48df78-403e-0251-3804-d0d1e2c6e762}" +2024-07-05T13:36:04.514Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:03.509429464 +0000 UTC m=+29.505350915 (1 nodes), 785.264031ms from start of poll, 788.087605ms from cluster start. Timeout at 2024-07-05 13:36:04.514526522 +0000 UTC m=+30.510447973, 1.005097058s after stability +2024-07-05T13:36:04.520Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: starting snapshot up to: index=12 +2024-07-05T13:36:04.520Z [INFO] creating new snapshot: path=/tmp/raft1410466743/snapshots/2-12-1720186564520.tmp +2024-07-05T13:36:04.524Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: snapshot complete up to: index=12 +2024-07-05T13:36:04.531Z [INFO] creating new snapshot: path=/tmp/raft1410466743/snapshots/2-23-1720186564531.tmp +2024-07-05T13:36:04.535Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: copied to local snapshot: bytes=71 +2024-07-05T13:36:04.537Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: snapshot restore progress: id=2-23-1720186564531 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:04.537Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: restored user snapshot: index=23 +2024-07-05T13:36:04.537Z [ERROR] server-5536b621-794f-ff76-6e28-056070cb1441: failed to get log: index=23 error="log not found" +2024-07-05T13:36:04.538Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: aborting pipeline replication: peer="{Voter server-f14e4729-c511-7e3e-7fd0-5f12d8cc2692 f14e4729-c511-7e3e-7fd0-5f12d8cc2692}" +2024-07-05T13:36:04.537Z [ERROR] server-5536b621-794f-ff76-6e28-056070cb1441: failed to get log: index=23 error="log not found" +2024-07-05T13:36:04.538Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: aborting pipeline replication: peer="{Voter server-dd48df78-403e-0251-3804-d0d1e2c6e762 dd48df78-403e-0251-3804-d0d1e2c6e762}" +2024-07-05T13:36:04.549Z [ERROR] server-5536b621-794f-ff76-6e28-056070cb1441: failed to get log: index=23 error="log not found" +2024-07-05T13:36:04.550Z [ERROR] server-5536b621-794f-ff76-6e28-056070cb1441: failed to get log: index=23 error="log not found" +2024-07-05T13:36:04.553Z [INFO] creating new snapshot: path=/tmp/raft749414882/snapshots/2-23-1720186564553.tmp +2024-07-05T13:36:04.554Z [INFO] creating new snapshot: path=/tmp/raft2880961825/snapshots/2-23-1720186564554.tmp +2024-07-05T13:36:04.555Z [INFO] server-f14e4729-c511-7e3e-7fd0-5f12d8cc2692: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:04.556Z [INFO] server-dd48df78-403e-0251-3804-d0d1e2c6e762: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:04.558Z [INFO] server-f14e4729-c511-7e3e-7fd0-5f12d8cc2692: copied to local snapshot: bytes=71 +2024-07-05T13:36:04.558Z [INFO] server-dd48df78-403e-0251-3804-d0d1e2c6e762: copied to local snapshot: bytes=71 +2024-07-05T13:36:04.559Z [INFO] server-f14e4729-c511-7e3e-7fd0-5f12d8cc2692: snapshot restore progress: id=2-23-1720186564553 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:04.560Z [INFO] server-f14e4729-c511-7e3e-7fd0-5f12d8cc2692: Installed remote snapshot +2024-07-05T13:36:04.560Z [INFO] server-dd48df78-403e-0251-3804-d0d1e2c6e762: snapshot restore progress: id=2-23-1720186564554 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:04.560Z [INFO] server-dd48df78-403e-0251-3804-d0d1e2c6e762: Installed remote snapshot +2024-07-05T13:36:04.560Z [WARN] server-f14e4729-c511-7e3e-7fd0-5f12d8cc2692: skipping application of old log: index=22 +2024-07-05T13:36:04.561Z [WARN] server-dd48df78-403e-0251-3804-d0d1e2c6e762: skipping application of old log: index=23 +2024-07-05T13:36:04.561Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: pipelining replication: peer="{Voter server-f14e4729-c511-7e3e-7fd0-5f12d8cc2692 f14e4729-c511-7e3e-7fd0-5f12d8cc2692}" +2024-07-05T13:36:04.561Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: pipelining replication: peer="{Voter server-dd48df78-403e-0251-3804-d0d1e2c6e762 dd48df78-403e-0251-3804-d0d1e2c6e762}" +2024-07-05T13:36:04.561Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:36:05.567Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:36:04.561874824 +0000 UTC m=+30.557795984 (2 nodes), 8.459µs from start of poll, 1.840532674s from cluster start. Timeout at 2024-07-05 13:36:05.567239918 +0000 UTC m=+31.563161370, 1.005365386s after stability +2024-07-05T13:36:05.584Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: aborting pipeline replication: peer="{Voter server-dd48df78-403e-0251-3804-d0d1e2c6e762 dd48df78-403e-0251-3804-d0d1e2c6e762}" +2024-07-05T13:36:05.584Z [INFO] server-5536b621-794f-ff76-6e28-056070cb1441: aborting pipeline replication: peer="{Voter server-f14e4729-c511-7e3e-7fd0-5f12d8cc2692 f14e4729-c511-7e3e-7fd0-5f12d8cc2692}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false -2024-07-05T13:29:49.602Z [INFO] server-d50f384f-de98-b821-7435-b54c0ff95742: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d50f384f-de98-b821-7435-b54c0ff95742 Address:d50f384f-de98-b821-7435-b54c0ff95742} {Suffrage:Voter ID:server-3909df40-3582-82cd-05c0-3b17af291f80 Address:3909df40-3582-82cd-05c0-3b17af291f80} {Suffrage:Voter ID:server-2f957615-c29b-b42e-261a-56d0b040226a Address:2f957615-c29b-b42e-261a-56d0b040226a}]" -2024-07-05T13:29:49.602Z [INFO] server-d50f384f-de98-b821-7435-b54c0ff95742: entering follower state: follower="Node at d50f384f-de98-b821-7435-b54c0ff95742 [Follower]" leader-address= leader-id= -2024-07-05T13:29:49.602Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d50f384f-de98-b821-7435-b54c0ff95742 Address:d50f384f-de98-b821-7435-b54c0ff95742} {Suffrage:Voter ID:server-3909df40-3582-82cd-05c0-3b17af291f80 Address:3909df40-3582-82cd-05c0-3b17af291f80} {Suffrage:Voter ID:server-2f957615-c29b-b42e-261a-56d0b040226a Address:2f957615-c29b-b42e-261a-56d0b040226a}]" -2024-07-05T13:29:49.603Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: entering follower state: follower="Node at 3909df40-3582-82cd-05c0-3b17af291f80 [Follower]" leader-address= leader-id= -2024-07-05T13:29:49.603Z [INFO] server-2f957615-c29b-b42e-261a-56d0b040226a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d50f384f-de98-b821-7435-b54c0ff95742 Address:d50f384f-de98-b821-7435-b54c0ff95742} {Suffrage:Voter ID:server-3909df40-3582-82cd-05c0-3b17af291f80 Address:3909df40-3582-82cd-05c0-3b17af291f80} {Suffrage:Voter ID:server-2f957615-c29b-b42e-261a-56d0b040226a Address:2f957615-c29b-b42e-261a-56d0b040226a}]" -2024-07-05T13:29:49.603Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:49.603Z [INFO] server-2f957615-c29b-b42e-261a-56d0b040226a: entering follower state: follower="Node at 2f957615-c29b-b42e-261a-56d0b040226a [Follower]" leader-address= leader-id= -2024-07-05T13:29:50.300Z [WARN] server-3909df40-3582-82cd-05c0-3b17af291f80: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:50.300Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: entering candidate state: node="Node at 3909df40-3582-82cd-05c0-3b17af291f80 [Candidate]" term=2 -2024-07-05T13:29:50.301Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: election won: term=2 tally=2 -2024-07-05T13:29:50.301Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: entering leader state: leader="Node at 3909df40-3582-82cd-05c0-3b17af291f80 [Leader]" -2024-07-05T13:29:50.301Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: added peer, starting replication: peer=server-d50f384f-de98-b821-7435-b54c0ff95742 -2024-07-05T13:29:50.301Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: added peer, starting replication: peer=server-2f957615-c29b-b42e-261a-56d0b040226a -2024-07-05T13:29:50.301Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: pipelining replication: peer="{Voter server-2f957615-c29b-b42e-261a-56d0b040226a 2f957615-c29b-b42e-261a-56d0b040226a}" -2024-07-05T13:29:50.301Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: pipelining replication: peer="{Voter server-d50f384f-de98-b821-7435-b54c0ff95742 d50f384f-de98-b821-7435-b54c0ff95742}" -2024-07-05T13:29:51.307Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:50.301635509 +0000 UTC m=+31.071546184 (1 nodes), 697.971274ms from start of poll, 699.23443ms from cluster start. Timeout at 2024-07-05 13:29:51.307063157 +0000 UTC m=+32.076973872, 1.005427688s after stability -2024-07-05T13:29:51.309Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: starting snapshot up to: index=12 -2024-07-05T13:29:51.309Z [INFO] creating new snapshot: path=/tmp/raft2056148802/snapshots/2-12-1720186191309.tmp -2024-07-05T13:29:51.310Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: snapshot complete up to: index=12 -2024-07-05T13:29:51.313Z [INFO] creating new snapshot: path=/tmp/raft2056148802/snapshots/2-23-1720186191313.tmp -2024-07-05T13:29:51.314Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: copied to local snapshot: bytes=71 -2024-07-05T13:29:51.314Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: snapshot restore progress: id=2-23-1720186191313 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:51.314Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: removing all old logs from log store -2024-07-05T13:29:51.314Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: compacting logs: from=1 to=22 -2024-07-05T13:29:51.314Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: restored user snapshot: index=23 -2024-07-05T13:29:51.314Z [ERROR] server-3909df40-3582-82cd-05c0-3b17af291f80: failed to get log: index=22 error="log not found" -2024-07-05T13:29:51.314Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: aborting pipeline replication: peer="{Voter server-2f957615-c29b-b42e-261a-56d0b040226a 2f957615-c29b-b42e-261a-56d0b040226a}" -2024-07-05T13:29:51.314Z [ERROR] server-3909df40-3582-82cd-05c0-3b17af291f80: failed to get log: index=22 error="log not found" -2024-07-05T13:29:51.314Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: aborting pipeline replication: peer="{Voter server-d50f384f-de98-b821-7435-b54c0ff95742 d50f384f-de98-b821-7435-b54c0ff95742}" -2024-07-05T13:29:51.325Z [ERROR] server-3909df40-3582-82cd-05c0-3b17af291f80: failed to get log: index=22 error="log not found" -2024-07-05T13:29:51.326Z [INFO] creating new snapshot: path=/tmp/raft848574519/snapshots/2-23-1720186191326.tmp -2024-07-05T13:29:51.329Z [ERROR] server-3909df40-3582-82cd-05c0-3b17af291f80: failed to get log: index=22 error="log not found" -2024-07-05T13:29:51.330Z [INFO] creating new snapshot: path=/tmp/raft805195852/snapshots/2-23-1720186191330.tmp -2024-07-05T13:29:51.331Z [INFO] server-2f957615-c29b-b42e-261a-56d0b040226a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:51.337Z [INFO] server-2f957615-c29b-b42e-261a-56d0b040226a: copied to local snapshot: bytes=71 -2024-07-05T13:29:51.337Z [INFO] server-2f957615-c29b-b42e-261a-56d0b040226a: snapshot restore progress: id=2-23-1720186191330 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:51.337Z [INFO] server-2f957615-c29b-b42e-261a-56d0b040226a: removing all old logs from log store -2024-07-05T13:29:51.337Z [INFO] server-2f957615-c29b-b42e-261a-56d0b040226a: compacting logs: from=1 to=22 -2024-07-05T13:29:51.338Z [INFO] server-2f957615-c29b-b42e-261a-56d0b040226a: Installed remote snapshot -2024-07-05T13:29:51.340Z [INFO] server-d50f384f-de98-b821-7435-b54c0ff95742: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:51.340Z [WARN] server-2f957615-c29b-b42e-261a-56d0b040226a: skipping application of old log: index=23 -2024-07-05T13:29:51.340Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: pipelining replication: peer="{Voter server-2f957615-c29b-b42e-261a-56d0b040226a 2f957615-c29b-b42e-261a-56d0b040226a}" -2024-07-05T13:29:51.340Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:29:51.340Z [INFO] server-d50f384f-de98-b821-7435-b54c0ff95742: copied to local snapshot: bytes=71 -2024-07-05T13:29:51.341Z [INFO] server-d50f384f-de98-b821-7435-b54c0ff95742: snapshot restore progress: id=2-23-1720186191326 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:51.341Z [INFO] server-d50f384f-de98-b821-7435-b54c0ff95742: removing all old logs from log store -2024-07-05T13:29:51.341Z [INFO] server-d50f384f-de98-b821-7435-b54c0ff95742: compacting logs: from=1 to=22 -2024-07-05T13:29:51.341Z [INFO] server-d50f384f-de98-b821-7435-b54c0ff95742: Installed remote snapshot -2024-07-05T13:29:51.341Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: pipelining replication: peer="{Voter server-d50f384f-de98-b821-7435-b54c0ff95742 d50f384f-de98-b821-7435-b54c0ff95742}" -2024-07-05T13:29:52.346Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:29:51.340809413 +0000 UTC m=+32.110720088 (2 nodes), 1.6µs from start of poll, 1.738408334s from cluster start. Timeout at 2024-07-05 13:29:52.345935832 +0000 UTC m=+33.115846507, 1.005126419s after stability -2024-07-05T13:29:52.364Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: aborting pipeline replication: peer="{Voter server-2f957615-c29b-b42e-261a-56d0b040226a 2f957615-c29b-b42e-261a-56d0b040226a}" -2024-07-05T13:29:52.364Z [INFO] server-3909df40-3582-82cd-05c0-3b17af291f80: aborting pipeline replication: peer="{Voter server-d50f384f-de98-b821-7435-b54c0ff95742 d50f384f-de98-b821-7435-b54c0ff95742}" +2024-07-05T13:36:05.600Z [INFO] server-63a30b1f-e75b-8550-7e49-d7439ebb87a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63a30b1f-e75b-8550-7e49-d7439ebb87a7 Address:63a30b1f-e75b-8550-7e49-d7439ebb87a7} {Suffrage:Voter ID:server-0e02c1be-3e32-8972-59db-9307a57dc71a Address:0e02c1be-3e32-8972-59db-9307a57dc71a} {Suffrage:Voter ID:server-66d3739a-6e65-d136-15a7-e61fbbc6088b Address:66d3739a-6e65-d136-15a7-e61fbbc6088b}]" +2024-07-05T13:36:05.600Z [INFO] server-63a30b1f-e75b-8550-7e49-d7439ebb87a7: entering follower state: follower="Node at 63a30b1f-e75b-8550-7e49-d7439ebb87a7 [Follower]" leader-address= leader-id= +2024-07-05T13:36:05.601Z [INFO] server-0e02c1be-3e32-8972-59db-9307a57dc71a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63a30b1f-e75b-8550-7e49-d7439ebb87a7 Address:63a30b1f-e75b-8550-7e49-d7439ebb87a7} {Suffrage:Voter ID:server-0e02c1be-3e32-8972-59db-9307a57dc71a Address:0e02c1be-3e32-8972-59db-9307a57dc71a} {Suffrage:Voter ID:server-66d3739a-6e65-d136-15a7-e61fbbc6088b Address:66d3739a-6e65-d136-15a7-e61fbbc6088b}]" +2024-07-05T13:36:05.601Z [INFO] server-0e02c1be-3e32-8972-59db-9307a57dc71a: entering follower state: follower="Node at 0e02c1be-3e32-8972-59db-9307a57dc71a [Follower]" leader-address= leader-id= +2024-07-05T13:36:05.602Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63a30b1f-e75b-8550-7e49-d7439ebb87a7 Address:63a30b1f-e75b-8550-7e49-d7439ebb87a7} {Suffrage:Voter ID:server-0e02c1be-3e32-8972-59db-9307a57dc71a Address:0e02c1be-3e32-8972-59db-9307a57dc71a} {Suffrage:Voter ID:server-66d3739a-6e65-d136-15a7-e61fbbc6088b Address:66d3739a-6e65-d136-15a7-e61fbbc6088b}]" +2024-07-05T13:36:05.602Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:05.602Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: entering follower state: follower="Node at 66d3739a-6e65-d136-15a7-e61fbbc6088b [Follower]" leader-address= leader-id= +2024-07-05T13:36:06.234Z [WARN] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:06.234Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: entering candidate state: node="Node at 66d3739a-6e65-d136-15a7-e61fbbc6088b [Candidate]" term=2 +2024-07-05T13:36:06.235Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: election won: term=2 tally=2 +2024-07-05T13:36:06.235Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: entering leader state: leader="Node at 66d3739a-6e65-d136-15a7-e61fbbc6088b [Leader]" +2024-07-05T13:36:06.235Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: added peer, starting replication: peer=server-63a30b1f-e75b-8550-7e49-d7439ebb87a7 +2024-07-05T13:36:06.235Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: added peer, starting replication: peer=server-0e02c1be-3e32-8972-59db-9307a57dc71a +2024-07-05T13:36:06.236Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: pipelining replication: peer="{Voter server-0e02c1be-3e32-8972-59db-9307a57dc71a 0e02c1be-3e32-8972-59db-9307a57dc71a}" +2024-07-05T13:36:06.236Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: pipelining replication: peer="{Voter server-63a30b1f-e75b-8550-7e49-d7439ebb87a7 63a30b1f-e75b-8550-7e49-d7439ebb87a7}" +2024-07-05T13:36:07.241Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:06.235504004 +0000 UTC m=+32.231425456 (1 nodes), 633.086812ms from start of poll, 636.149839ms from cluster start. Timeout at 2024-07-05 13:36:07.241359965 +0000 UTC m=+33.237281417, 1.005855961s after stability +2024-07-05T13:36:07.246Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: starting snapshot up to: index=12 +2024-07-05T13:36:07.247Z [INFO] creating new snapshot: path=/tmp/raft27568096/snapshots/2-12-1720186567246.tmp +2024-07-05T13:36:07.251Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: snapshot complete up to: index=12 +2024-07-05T13:36:07.262Z [INFO] creating new snapshot: path=/tmp/raft27568096/snapshots/2-23-1720186567262.tmp +2024-07-05T13:36:07.268Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: copied to local snapshot: bytes=71 +2024-07-05T13:36:07.269Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: snapshot restore progress: id=2-23-1720186567262 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:07.270Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: removing all old logs from log store +2024-07-05T13:36:07.270Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: compacting logs: from=1 to=22 +2024-07-05T13:36:07.270Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: restored user snapshot: index=23 +2024-07-05T13:36:07.270Z [ERROR] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: failed to get log: index=22 error="log not found" +2024-07-05T13:36:07.270Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: aborting pipeline replication: peer="{Voter server-63a30b1f-e75b-8550-7e49-d7439ebb87a7 63a30b1f-e75b-8550-7e49-d7439ebb87a7}" +2024-07-05T13:36:07.270Z [ERROR] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: failed to get log: index=22 error="log not found" +2024-07-05T13:36:07.271Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: aborting pipeline replication: peer="{Voter server-0e02c1be-3e32-8972-59db-9307a57dc71a 0e02c1be-3e32-8972-59db-9307a57dc71a}" +2024-07-05T13:36:07.276Z [ERROR] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: failed to get log: index=22 error="log not found" +2024-07-05T13:36:07.278Z [INFO] creating new snapshot: path=/tmp/raft3970949251/snapshots/2-23-1720186567278.tmp +2024-07-05T13:36:07.279Z [ERROR] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: failed to get log: index=22 error="log not found" +2024-07-05T13:36:07.280Z [INFO] server-63a30b1f-e75b-8550-7e49-d7439ebb87a7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:07.281Z [INFO] creating new snapshot: path=/tmp/raft1455205065/snapshots/2-23-1720186567281.tmp +2024-07-05T13:36:07.282Z [INFO] server-63a30b1f-e75b-8550-7e49-d7439ebb87a7: copied to local snapshot: bytes=71 +2024-07-05T13:36:07.283Z [INFO] server-0e02c1be-3e32-8972-59db-9307a57dc71a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:07.283Z [INFO] server-63a30b1f-e75b-8550-7e49-d7439ebb87a7: snapshot restore progress: id=2-23-1720186567278 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:07.283Z [INFO] server-63a30b1f-e75b-8550-7e49-d7439ebb87a7: removing all old logs from log store +2024-07-05T13:36:07.283Z [INFO] server-63a30b1f-e75b-8550-7e49-d7439ebb87a7: compacting logs: from=1 to=22 +2024-07-05T13:36:07.283Z [INFO] server-63a30b1f-e75b-8550-7e49-d7439ebb87a7: Installed remote snapshot +2024-07-05T13:36:07.284Z [WARN] server-63a30b1f-e75b-8550-7e49-d7439ebb87a7: skipping application of old log: index=23 +2024-07-05T13:36:07.285Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: pipelining replication: peer="{Voter server-63a30b1f-e75b-8550-7e49-d7439ebb87a7 63a30b1f-e75b-8550-7e49-d7439ebb87a7}" +2024-07-05T13:36:07.285Z [INFO] server-0e02c1be-3e32-8972-59db-9307a57dc71a: copied to local snapshot: bytes=71 +2024-07-05T13:36:07.285Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:36:07.287Z [INFO] server-0e02c1be-3e32-8972-59db-9307a57dc71a: snapshot restore progress: id=2-23-1720186567281 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:07.287Z [INFO] server-0e02c1be-3e32-8972-59db-9307a57dc71a: removing all old logs from log store +2024-07-05T13:36:07.287Z [INFO] server-0e02c1be-3e32-8972-59db-9307a57dc71a: compacting logs: from=1 to=22 +2024-07-05T13:36:07.287Z [INFO] server-0e02c1be-3e32-8972-59db-9307a57dc71a: Installed remote snapshot +2024-07-05T13:36:07.288Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: pipelining replication: peer="{Voter server-0e02c1be-3e32-8972-59db-9307a57dc71a 0e02c1be-3e32-8972-59db-9307a57dc71a}" +2024-07-05T13:36:08.291Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:36:07.285991441 +0000 UTC m=+33.281912893 (2 nodes), 8.458µs from start of poll, 1.686637276s from cluster start. Timeout at 2024-07-05 13:36:08.291391827 +0000 UTC m=+34.287313279, 1.005400386s after stability +2024-07-05T13:36:08.307Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: aborting pipeline replication: peer="{Voter server-0e02c1be-3e32-8972-59db-9307a57dc71a 0e02c1be-3e32-8972-59db-9307a57dc71a}" +2024-07-05T13:36:08.307Z [INFO] server-66d3739a-6e65-d136-15a7-e61fbbc6088b: aborting pipeline replication: peer="{Voter server-63a30b1f-e75b-8550-7e49-d7439ebb87a7 63a30b1f-e75b-8550-7e49-d7439ebb87a7}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true -2024-07-05T13:29:52.367Z [INFO] server-c70a7859-3767-b8d7-5ad8-562c4c33faf0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c70a7859-3767-b8d7-5ad8-562c4c33faf0 Address:c70a7859-3767-b8d7-5ad8-562c4c33faf0} {Suffrage:Voter ID:server-d95465fd-a04d-dff5-c54d-1f185452383d Address:d95465fd-a04d-dff5-c54d-1f185452383d} {Suffrage:Voter ID:server-93ec1b36-f09c-77e4-8e76-50de1e57427c Address:93ec1b36-f09c-77e4-8e76-50de1e57427c}]" -2024-07-05T13:29:52.368Z [INFO] server-c70a7859-3767-b8d7-5ad8-562c4c33faf0: entering follower state: follower="Node at c70a7859-3767-b8d7-5ad8-562c4c33faf0 [Follower]" leader-address= leader-id= -2024-07-05T13:29:52.368Z [INFO] server-d95465fd-a04d-dff5-c54d-1f185452383d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c70a7859-3767-b8d7-5ad8-562c4c33faf0 Address:c70a7859-3767-b8d7-5ad8-562c4c33faf0} {Suffrage:Voter ID:server-d95465fd-a04d-dff5-c54d-1f185452383d Address:d95465fd-a04d-dff5-c54d-1f185452383d} {Suffrage:Voter ID:server-93ec1b36-f09c-77e4-8e76-50de1e57427c Address:93ec1b36-f09c-77e4-8e76-50de1e57427c}]" -2024-07-05T13:29:52.375Z [INFO] server-93ec1b36-f09c-77e4-8e76-50de1e57427c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c70a7859-3767-b8d7-5ad8-562c4c33faf0 Address:c70a7859-3767-b8d7-5ad8-562c4c33faf0} {Suffrage:Voter ID:server-d95465fd-a04d-dff5-c54d-1f185452383d Address:d95465fd-a04d-dff5-c54d-1f185452383d} {Suffrage:Voter ID:server-93ec1b36-f09c-77e4-8e76-50de1e57427c Address:93ec1b36-f09c-77e4-8e76-50de1e57427c}]" -2024-07-05T13:29:52.375Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:52.377Z [INFO] server-d95465fd-a04d-dff5-c54d-1f185452383d: entering follower state: follower="Node at d95465fd-a04d-dff5-c54d-1f185452383d [Follower]" leader-address= leader-id= -2024-07-05T13:29:52.377Z [INFO] server-93ec1b36-f09c-77e4-8e76-50de1e57427c: entering follower state: follower="Node at 93ec1b36-f09c-77e4-8e76-50de1e57427c [Follower]" leader-address= leader-id= -2024-07-05T13:29:52.895Z [WARN] server-d95465fd-a04d-dff5-c54d-1f185452383d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:52.895Z [INFO] server-d95465fd-a04d-dff5-c54d-1f185452383d: entering candidate state: node="Node at d95465fd-a04d-dff5-c54d-1f185452383d [Candidate]" term=2 -2024-07-05T13:29:52.896Z [INFO] server-d95465fd-a04d-dff5-c54d-1f185452383d: election won: term=2 tally=2 -2024-07-05T13:29:52.896Z [INFO] server-d95465fd-a04d-dff5-c54d-1f185452383d: entering leader state: leader="Node at d95465fd-a04d-dff5-c54d-1f185452383d [Leader]" -2024-07-05T13:29:52.896Z [INFO] server-d95465fd-a04d-dff5-c54d-1f185452383d: added peer, starting replication: peer=server-c70a7859-3767-b8d7-5ad8-562c4c33faf0 -2024-07-05T13:29:52.896Z [INFO] server-d95465fd-a04d-dff5-c54d-1f185452383d: added peer, starting replication: peer=server-93ec1b36-f09c-77e4-8e76-50de1e57427c -2024-07-05T13:29:52.896Z [INFO] server-d95465fd-a04d-dff5-c54d-1f185452383d: pipelining replication: peer="{Voter server-93ec1b36-f09c-77e4-8e76-50de1e57427c 93ec1b36-f09c-77e4-8e76-50de1e57427c}" -2024-07-05T13:29:52.897Z [INFO] server-d95465fd-a04d-dff5-c54d-1f185452383d: pipelining replication: peer="{Voter server-c70a7859-3767-b8d7-5ad8-562c4c33faf0 c70a7859-3767-b8d7-5ad8-562c4c33faf0}" -2024-07-05T13:29:53.902Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:52.89698034 +0000 UTC m=+33.666891015 (1 nodes), 521.231514ms from start of poll, 529.861571ms from cluster start. Timeout at 2024-07-05 13:29:53.902488945 +0000 UTC m=+34.672399620, 1.005508605s after stability -2024-07-05T13:29:53.903Z [INFO] server-d95465fd-a04d-dff5-c54d-1f185452383d: starting snapshot up to: index=12 -2024-07-05T13:29:53.903Z [INFO] creating new snapshot: path=/tmp/raft3332081195/snapshots/2-12-1720186193903.tmp -2024-07-05T13:29:53.904Z [INFO] server-d95465fd-a04d-dff5-c54d-1f185452383d: snapshot complete up to: index=12 -2024-07-05T13:29:53.910Z [INFO] server-5dca7b22-c554-a416-4bbf-27f97f088122: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5dca7b22-c554-a416-4bbf-27f97f088122 Address:5dca7b22-c554-a416-4bbf-27f97f088122} {Suffrage:Voter ID:server-ca001b46-446f-f5f8-e393-8c8fb148c46c Address:ca001b46-446f-f5f8-e393-8c8fb148c46c} {Suffrage:Voter ID:server-fcb1fc29-a903-27d4-d725-162dc6290e59 Address:fcb1fc29-a903-27d4-d725-162dc6290e59}]" -2024-07-05T13:29:53.910Z [INFO] server-ca001b46-446f-f5f8-e393-8c8fb148c46c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5dca7b22-c554-a416-4bbf-27f97f088122 Address:5dca7b22-c554-a416-4bbf-27f97f088122} {Suffrage:Voter ID:server-ca001b46-446f-f5f8-e393-8c8fb148c46c Address:ca001b46-446f-f5f8-e393-8c8fb148c46c} {Suffrage:Voter ID:server-fcb1fc29-a903-27d4-d725-162dc6290e59 Address:fcb1fc29-a903-27d4-d725-162dc6290e59}]" -2024-07-05T13:29:53.910Z [INFO] server-fcb1fc29-a903-27d4-d725-162dc6290e59: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5dca7b22-c554-a416-4bbf-27f97f088122 Address:5dca7b22-c554-a416-4bbf-27f97f088122} {Suffrage:Voter ID:server-ca001b46-446f-f5f8-e393-8c8fb148c46c Address:ca001b46-446f-f5f8-e393-8c8fb148c46c} {Suffrage:Voter ID:server-fcb1fc29-a903-27d4-d725-162dc6290e59 Address:fcb1fc29-a903-27d4-d725-162dc6290e59}]" -2024-07-05T13:29:53.910Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:53.910Z [INFO] server-5dca7b22-c554-a416-4bbf-27f97f088122: entering follower state: follower="Node at 5dca7b22-c554-a416-4bbf-27f97f088122 [Follower]" leader-address= leader-id= -2024-07-05T13:29:53.910Z [INFO] server-ca001b46-446f-f5f8-e393-8c8fb148c46c: entering follower state: follower="Node at ca001b46-446f-f5f8-e393-8c8fb148c46c [Follower]" leader-address= leader-id= -2024-07-05T13:29:53.910Z [INFO] server-fcb1fc29-a903-27d4-d725-162dc6290e59: entering follower state: follower="Node at fcb1fc29-a903-27d4-d725-162dc6290e59 [Follower]" leader-address= leader-id= -2024-07-05T13:29:54.436Z [WARN] server-5dca7b22-c554-a416-4bbf-27f97f088122: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:54.436Z [INFO] server-5dca7b22-c554-a416-4bbf-27f97f088122: entering candidate state: node="Node at 5dca7b22-c554-a416-4bbf-27f97f088122 [Candidate]" term=2 -2024-07-05T13:29:54.436Z [INFO] server-5dca7b22-c554-a416-4bbf-27f97f088122: election won: term=2 tally=2 -2024-07-05T13:29:54.436Z [INFO] server-5dca7b22-c554-a416-4bbf-27f97f088122: entering leader state: leader="Node at 5dca7b22-c554-a416-4bbf-27f97f088122 [Leader]" -2024-07-05T13:29:54.436Z [INFO] server-5dca7b22-c554-a416-4bbf-27f97f088122: added peer, starting replication: peer=server-ca001b46-446f-f5f8-e393-8c8fb148c46c -2024-07-05T13:29:54.436Z [INFO] server-5dca7b22-c554-a416-4bbf-27f97f088122: added peer, starting replication: peer=server-fcb1fc29-a903-27d4-d725-162dc6290e59 -2024-07-05T13:29:54.436Z [INFO] server-5dca7b22-c554-a416-4bbf-27f97f088122: pipelining replication: peer="{Voter server-fcb1fc29-a903-27d4-d725-162dc6290e59 fcb1fc29-a903-27d4-d725-162dc6290e59}" -2024-07-05T13:29:54.436Z [INFO] server-5dca7b22-c554-a416-4bbf-27f97f088122: pipelining replication: peer="{Voter server-ca001b46-446f-f5f8-e393-8c8fb148c46c ca001b46-446f-f5f8-e393-8c8fb148c46c}" -2024-07-05T13:29:55.442Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:54.437003434 +0000 UTC m=+35.206914109 (1 nodes), 526.318976ms from start of poll, 527.472256ms from cluster start. Timeout at 2024-07-05 13:29:55.442093293 +0000 UTC m=+36.212003968, 1.005089859s after stability -2024-07-05T13:29:55.442Z [INFO] creating new snapshot: path=/tmp/raft2535160448/snapshots/2-14-1720186195442.tmp -2024-07-05T13:29:55.443Z [INFO] server-5dca7b22-c554-a416-4bbf-27f97f088122: copied to local snapshot: bytes=71 -2024-07-05T13:29:55.443Z [INFO] server-5dca7b22-c554-a416-4bbf-27f97f088122: snapshot restore progress: id=2-14-1720186195442 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:55.444Z [INFO] server-5dca7b22-c554-a416-4bbf-27f97f088122: restored user snapshot: index=14 -2024-07-05T13:29:55.444Z [ERROR] server-5dca7b22-c554-a416-4bbf-27f97f088122: failed to get log: index=3 error="log not found" -2024-07-05T13:29:55.444Z [INFO] server-5dca7b22-c554-a416-4bbf-27f97f088122: aborting pipeline replication: peer="{Voter server-fcb1fc29-a903-27d4-d725-162dc6290e59 fcb1fc29-a903-27d4-d725-162dc6290e59}" -2024-07-05T13:29:55.444Z [ERROR] server-5dca7b22-c554-a416-4bbf-27f97f088122: failed to get log: index=3 error="log not found" -2024-07-05T13:29:55.444Z [INFO] server-5dca7b22-c554-a416-4bbf-27f97f088122: aborting pipeline replication: peer="{Voter server-ca001b46-446f-f5f8-e393-8c8fb148c46c ca001b46-446f-f5f8-e393-8c8fb148c46c}" -2024-07-05T13:29:55.450Z [ERROR] server-5dca7b22-c554-a416-4bbf-27f97f088122: failed to get log: index=3 error="log not found" -2024-07-05T13:29:55.450Z [INFO] creating new snapshot: path=/tmp/raft2528897045/snapshots/2-14-1720186195450.tmp -2024-07-05T13:29:55.451Z [INFO] server-ca001b46-446f-f5f8-e393-8c8fb148c46c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:55.451Z [INFO] server-ca001b46-446f-f5f8-e393-8c8fb148c46c: copied to local snapshot: bytes=71 -2024-07-05T13:29:55.452Z [INFO] server-ca001b46-446f-f5f8-e393-8c8fb148c46c: snapshot restore progress: id=2-14-1720186195450 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:55.452Z [INFO] server-ca001b46-446f-f5f8-e393-8c8fb148c46c: Installed remote snapshot -2024-07-05T13:29:55.452Z [INFO] server-5dca7b22-c554-a416-4bbf-27f97f088122: pipelining replication: peer="{Voter server-ca001b46-446f-f5f8-e393-8c8fb148c46c ca001b46-446f-f5f8-e393-8c8fb148c46c}" -2024-07-05T13:29:55.452Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:29:55.454Z [ERROR] server-5dca7b22-c554-a416-4bbf-27f97f088122: failed to get log: index=3 error="log not found" -2024-07-05T13:29:55.454Z [INFO] creating new snapshot: path=/tmp/raft2117437642/snapshots/2-14-1720186195454.tmp -2024-07-05T13:29:55.455Z [INFO] server-fcb1fc29-a903-27d4-d725-162dc6290e59: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:55.455Z [INFO] server-fcb1fc29-a903-27d4-d725-162dc6290e59: copied to local snapshot: bytes=71 -2024-07-05T13:29:55.461Z [INFO] server-fcb1fc29-a903-27d4-d725-162dc6290e59: snapshot restore progress: id=2-14-1720186195454 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:55.461Z [INFO] server-fcb1fc29-a903-27d4-d725-162dc6290e59: Installed remote snapshot -2024-07-05T13:29:55.461Z [INFO] server-5dca7b22-c554-a416-4bbf-27f97f088122: pipelining replication: peer="{Voter server-fcb1fc29-a903-27d4-d725-162dc6290e59 fcb1fc29-a903-27d4-d725-162dc6290e59}" -2024-07-05T13:29:56.458Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:29:55.452796038 +0000 UTC m=+36.222706713 (2 nodes), 1.64µs from start of poll, 1.54326486s from cluster start. Timeout at 2024-07-05 13:29:56.45809881 +0000 UTC m=+37.228009485, 1.005302772s after stability -2024-07-05T13:29:56.471Z [INFO] server-d95465fd-a04d-dff5-c54d-1f185452383d: aborting pipeline replication: peer="{Voter server-93ec1b36-f09c-77e4-8e76-50de1e57427c 93ec1b36-f09c-77e4-8e76-50de1e57427c}" -2024-07-05T13:29:56.471Z [INFO] server-d95465fd-a04d-dff5-c54d-1f185452383d: aborting pipeline replication: peer="{Voter server-c70a7859-3767-b8d7-5ad8-562c4c33faf0 c70a7859-3767-b8d7-5ad8-562c4c33faf0}" +2024-07-05T13:36:08.322Z [INFO] server-fa9fafbf-50d0-fee3-9994-1f417ae03903: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa9fafbf-50d0-fee3-9994-1f417ae03903 Address:fa9fafbf-50d0-fee3-9994-1f417ae03903} {Suffrage:Voter ID:server-ef39d93c-cc4c-e4fc-9568-e64056bb2e22 Address:ef39d93c-cc4c-e4fc-9568-e64056bb2e22} {Suffrage:Voter ID:server-a4da8f11-fd66-0551-a6b1-e561335ef5de Address:a4da8f11-fd66-0551-a6b1-e561335ef5de}]" +2024-07-05T13:36:08.322Z [INFO] server-fa9fafbf-50d0-fee3-9994-1f417ae03903: entering follower state: follower="Node at fa9fafbf-50d0-fee3-9994-1f417ae03903 [Follower]" leader-address= leader-id= +2024-07-05T13:36:08.323Z [INFO] server-ef39d93c-cc4c-e4fc-9568-e64056bb2e22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa9fafbf-50d0-fee3-9994-1f417ae03903 Address:fa9fafbf-50d0-fee3-9994-1f417ae03903} {Suffrage:Voter ID:server-ef39d93c-cc4c-e4fc-9568-e64056bb2e22 Address:ef39d93c-cc4c-e4fc-9568-e64056bb2e22} {Suffrage:Voter ID:server-a4da8f11-fd66-0551-a6b1-e561335ef5de Address:a4da8f11-fd66-0551-a6b1-e561335ef5de}]" +2024-07-05T13:36:08.323Z [INFO] server-ef39d93c-cc4c-e4fc-9568-e64056bb2e22: entering follower state: follower="Node at ef39d93c-cc4c-e4fc-9568-e64056bb2e22 [Follower]" leader-address= leader-id= +2024-07-05T13:36:08.324Z [INFO] server-a4da8f11-fd66-0551-a6b1-e561335ef5de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa9fafbf-50d0-fee3-9994-1f417ae03903 Address:fa9fafbf-50d0-fee3-9994-1f417ae03903} {Suffrage:Voter ID:server-ef39d93c-cc4c-e4fc-9568-e64056bb2e22 Address:ef39d93c-cc4c-e4fc-9568-e64056bb2e22} {Suffrage:Voter ID:server-a4da8f11-fd66-0551-a6b1-e561335ef5de Address:a4da8f11-fd66-0551-a6b1-e561335ef5de}]" +2024-07-05T13:36:08.324Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:08.324Z [INFO] server-a4da8f11-fd66-0551-a6b1-e561335ef5de: entering follower state: follower="Node at a4da8f11-fd66-0551-a6b1-e561335ef5de [Follower]" leader-address= leader-id= +2024-07-05T13:36:08.925Z [WARN] server-fa9fafbf-50d0-fee3-9994-1f417ae03903: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:08.925Z [INFO] server-fa9fafbf-50d0-fee3-9994-1f417ae03903: entering candidate state: node="Node at fa9fafbf-50d0-fee3-9994-1f417ae03903 [Candidate]" term=2 +2024-07-05T13:36:08.926Z [INFO] server-fa9fafbf-50d0-fee3-9994-1f417ae03903: election won: term=2 tally=2 +2024-07-05T13:36:08.926Z [INFO] server-fa9fafbf-50d0-fee3-9994-1f417ae03903: entering leader state: leader="Node at fa9fafbf-50d0-fee3-9994-1f417ae03903 [Leader]" +2024-07-05T13:36:08.926Z [INFO] server-fa9fafbf-50d0-fee3-9994-1f417ae03903: added peer, starting replication: peer=server-ef39d93c-cc4c-e4fc-9568-e64056bb2e22 +2024-07-05T13:36:08.926Z [INFO] server-fa9fafbf-50d0-fee3-9994-1f417ae03903: added peer, starting replication: peer=server-a4da8f11-fd66-0551-a6b1-e561335ef5de +2024-07-05T13:36:08.927Z [INFO] server-fa9fafbf-50d0-fee3-9994-1f417ae03903: pipelining replication: peer="{Voter server-a4da8f11-fd66-0551-a6b1-e561335ef5de a4da8f11-fd66-0551-a6b1-e561335ef5de}" +2024-07-05T13:36:08.928Z [INFO] server-fa9fafbf-50d0-fee3-9994-1f417ae03903: pipelining replication: peer="{Voter server-ef39d93c-cc4c-e4fc-9568-e64056bb2e22 ef39d93c-cc4c-e4fc-9568-e64056bb2e22}" +2024-07-05T13:36:09.932Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:08.926591725 +0000 UTC m=+34.922513177 (1 nodes), 601.871757ms from start of poll, 604.71283ms from cluster start. Timeout at 2024-07-05 13:36:09.931890322 +0000 UTC m=+35.927811773, 1.005298596s after stability +2024-07-05T13:36:09.945Z [INFO] server-fa9fafbf-50d0-fee3-9994-1f417ae03903: starting snapshot up to: index=12 +2024-07-05T13:36:09.945Z [INFO] creating new snapshot: path=/tmp/raft2413999342/snapshots/2-12-1720186569945.tmp +2024-07-05T13:36:09.949Z [INFO] server-fa9fafbf-50d0-fee3-9994-1f417ae03903: snapshot complete up to: index=12 +2024-07-05T13:36:09.960Z [INFO] server-832c0dba-b955-4596-20bd-842a7ba0a28a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-832c0dba-b955-4596-20bd-842a7ba0a28a Address:832c0dba-b955-4596-20bd-842a7ba0a28a} {Suffrage:Voter ID:server-cf97e764-091c-ecf1-61bd-929bf6bfee73 Address:cf97e764-091c-ecf1-61bd-929bf6bfee73} {Suffrage:Voter ID:server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3 Address:ce1cfe64-dcbf-fc0f-2642-b4a4613397b3}]" +2024-07-05T13:36:09.961Z [INFO] server-832c0dba-b955-4596-20bd-842a7ba0a28a: entering follower state: follower="Node at 832c0dba-b955-4596-20bd-842a7ba0a28a [Follower]" leader-address= leader-id= +2024-07-05T13:36:09.961Z [INFO] server-cf97e764-091c-ecf1-61bd-929bf6bfee73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-832c0dba-b955-4596-20bd-842a7ba0a28a Address:832c0dba-b955-4596-20bd-842a7ba0a28a} {Suffrage:Voter ID:server-cf97e764-091c-ecf1-61bd-929bf6bfee73 Address:cf97e764-091c-ecf1-61bd-929bf6bfee73} {Suffrage:Voter ID:server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3 Address:ce1cfe64-dcbf-fc0f-2642-b4a4613397b3}]" +2024-07-05T13:36:09.961Z [INFO] server-cf97e764-091c-ecf1-61bd-929bf6bfee73: entering follower state: follower="Node at cf97e764-091c-ecf1-61bd-929bf6bfee73 [Follower]" leader-address= leader-id= +2024-07-05T13:36:09.962Z [INFO] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-832c0dba-b955-4596-20bd-842a7ba0a28a Address:832c0dba-b955-4596-20bd-842a7ba0a28a} {Suffrage:Voter ID:server-cf97e764-091c-ecf1-61bd-929bf6bfee73 Address:cf97e764-091c-ecf1-61bd-929bf6bfee73} {Suffrage:Voter ID:server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3 Address:ce1cfe64-dcbf-fc0f-2642-b4a4613397b3}]" +2024-07-05T13:36:09.962Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:09.962Z [INFO] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: entering follower state: follower="Node at ce1cfe64-dcbf-fc0f-2642-b4a4613397b3 [Follower]" leader-address= leader-id= +2024-07-05T13:36:10.484Z [WARN] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:10.484Z [INFO] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: entering candidate state: node="Node at ce1cfe64-dcbf-fc0f-2642-b4a4613397b3 [Candidate]" term=2 +2024-07-05T13:36:10.485Z [INFO] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: election won: term=2 tally=2 +2024-07-05T13:36:10.485Z [INFO] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: entering leader state: leader="Node at ce1cfe64-dcbf-fc0f-2642-b4a4613397b3 [Leader]" +2024-07-05T13:36:10.486Z [INFO] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: added peer, starting replication: peer=server-832c0dba-b955-4596-20bd-842a7ba0a28a +2024-07-05T13:36:10.486Z [INFO] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: added peer, starting replication: peer=server-cf97e764-091c-ecf1-61bd-929bf6bfee73 +2024-07-05T13:36:10.487Z [INFO] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: pipelining replication: peer="{Voter server-832c0dba-b955-4596-20bd-842a7ba0a28a 832c0dba-b955-4596-20bd-842a7ba0a28a}" +2024-07-05T13:36:10.487Z [INFO] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: pipelining replication: peer="{Voter server-cf97e764-091c-ecf1-61bd-929bf6bfee73 cf97e764-091c-ecf1-61bd-929bf6bfee73}" +2024-07-05T13:36:11.491Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:10.486167323 +0000 UTC m=+36.482088774 (1 nodes), 523.564478ms from start of poll, 526.437634ms from cluster start. Timeout at 2024-07-05 13:36:11.491202841 +0000 UTC m=+37.487124292, 1.005035518s after stability +2024-07-05T13:36:11.492Z [INFO] creating new snapshot: path=/tmp/raft2798017595/snapshots/2-14-1720186571492.tmp +2024-07-05T13:36:11.497Z [INFO] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: copied to local snapshot: bytes=71 +2024-07-05T13:36:11.498Z [INFO] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: snapshot restore progress: id=2-14-1720186571492 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:11.499Z [INFO] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: restored user snapshot: index=14 +2024-07-05T13:36:11.499Z [ERROR] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: failed to get log: index=3 error="log not found" +2024-07-05T13:36:11.499Z [ERROR] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: failed to get log: index=3 error="log not found" +2024-07-05T13:36:11.499Z [INFO] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: aborting pipeline replication: peer="{Voter server-cf97e764-091c-ecf1-61bd-929bf6bfee73 cf97e764-091c-ecf1-61bd-929bf6bfee73}" +2024-07-05T13:36:11.500Z [INFO] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: aborting pipeline replication: peer="{Voter server-832c0dba-b955-4596-20bd-842a7ba0a28a 832c0dba-b955-4596-20bd-842a7ba0a28a}" +2024-07-05T13:36:11.507Z [ERROR] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: failed to get log: index=3 error="log not found" +2024-07-05T13:36:11.507Z [ERROR] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: failed to get log: index=3 error="log not found" +2024-07-05T13:36:11.509Z [INFO] creating new snapshot: path=/tmp/raft44228341/snapshots/2-14-1720186571509.tmp +2024-07-05T13:36:11.510Z [INFO] creating new snapshot: path=/tmp/raft431952940/snapshots/2-14-1720186571510.tmp +2024-07-05T13:36:11.511Z [INFO] server-832c0dba-b955-4596-20bd-842a7ba0a28a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:11.511Z [INFO] server-cf97e764-091c-ecf1-61bd-929bf6bfee73: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:11.517Z [INFO] server-cf97e764-091c-ecf1-61bd-929bf6bfee73: copied to local snapshot: bytes=71 +2024-07-05T13:36:11.523Z [INFO] server-832c0dba-b955-4596-20bd-842a7ba0a28a: copied to local snapshot: bytes=71 +2024-07-05T13:36:11.525Z [INFO] server-832c0dba-b955-4596-20bd-842a7ba0a28a: snapshot restore progress: id=2-14-1720186571509 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:11.525Z [INFO] server-cf97e764-091c-ecf1-61bd-929bf6bfee73: snapshot restore progress: id=2-14-1720186571510 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:11.525Z [INFO] server-832c0dba-b955-4596-20bd-842a7ba0a28a: Installed remote snapshot +2024-07-05T13:36:11.525Z [INFO] server-cf97e764-091c-ecf1-61bd-929bf6bfee73: Installed remote snapshot +2024-07-05T13:36:11.527Z [INFO] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: pipelining replication: peer="{Voter server-cf97e764-091c-ecf1-61bd-929bf6bfee73 cf97e764-091c-ecf1-61bd-929bf6bfee73}" +2024-07-05T13:36:11.527Z [INFO] server-ce1cfe64-dcbf-fc0f-2642-b4a4613397b3: pipelining replication: peer="{Voter server-832c0dba-b955-4596-20bd-842a7ba0a28a 832c0dba-b955-4596-20bd-842a7ba0a28a}" +2024-07-05T13:36:11.527Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:36:12.542Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:36:11.527442346 +0000 UTC m=+37.523363797 (2 nodes), 7.875µs from start of poll, 1.567712657s from cluster start. Timeout at 2024-07-05 13:36:12.53666437 +0000 UTC m=+38.532585822, 1.009222025s after stability +2024-07-05T13:36:12.558Z [INFO] server-fa9fafbf-50d0-fee3-9994-1f417ae03903: aborting pipeline replication: peer="{Voter server-a4da8f11-fd66-0551-a6b1-e561335ef5de a4da8f11-fd66-0551-a6b1-e561335ef5de}" +2024-07-05T13:36:12.559Z [INFO] server-fa9fafbf-50d0-fee3-9994-1f417ae03903: aborting pipeline replication: peer="{Voter server-ef39d93c-cc4c-e4fc-9568-e64056bb2e22 ef39d93c-cc4c-e4fc-9568-e64056bb2e22}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true -2024-07-05T13:29:56.473Z [INFO] server-2a7d3b39-116d-43a6-8c30-6669edfeb837: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2a7d3b39-116d-43a6-8c30-6669edfeb837 Address:2a7d3b39-116d-43a6-8c30-6669edfeb837} {Suffrage:Voter ID:server-ee87242f-8340-62b7-d0ce-d0cb2c67894e Address:ee87242f-8340-62b7-d0ce-d0cb2c67894e} {Suffrage:Voter ID:server-3d248845-0412-bac6-239d-8fd9e990a7c9 Address:3d248845-0412-bac6-239d-8fd9e990a7c9}]" -2024-07-05T13:29:56.474Z [INFO] server-2a7d3b39-116d-43a6-8c30-6669edfeb837: entering follower state: follower="Node at 2a7d3b39-116d-43a6-8c30-6669edfeb837 [Follower]" leader-address= leader-id= -2024-07-05T13:29:56.474Z [INFO] server-ee87242f-8340-62b7-d0ce-d0cb2c67894e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2a7d3b39-116d-43a6-8c30-6669edfeb837 Address:2a7d3b39-116d-43a6-8c30-6669edfeb837} {Suffrage:Voter ID:server-ee87242f-8340-62b7-d0ce-d0cb2c67894e Address:ee87242f-8340-62b7-d0ce-d0cb2c67894e} {Suffrage:Voter ID:server-3d248845-0412-bac6-239d-8fd9e990a7c9 Address:3d248845-0412-bac6-239d-8fd9e990a7c9}]" -2024-07-05T13:29:56.474Z [INFO] server-ee87242f-8340-62b7-d0ce-d0cb2c67894e: entering follower state: follower="Node at ee87242f-8340-62b7-d0ce-d0cb2c67894e [Follower]" leader-address= leader-id= -2024-07-05T13:29:56.474Z [INFO] server-3d248845-0412-bac6-239d-8fd9e990a7c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2a7d3b39-116d-43a6-8c30-6669edfeb837 Address:2a7d3b39-116d-43a6-8c30-6669edfeb837} {Suffrage:Voter ID:server-ee87242f-8340-62b7-d0ce-d0cb2c67894e Address:ee87242f-8340-62b7-d0ce-d0cb2c67894e} {Suffrage:Voter ID:server-3d248845-0412-bac6-239d-8fd9e990a7c9 Address:3d248845-0412-bac6-239d-8fd9e990a7c9}]" -2024-07-05T13:29:56.474Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:56.474Z [INFO] server-3d248845-0412-bac6-239d-8fd9e990a7c9: entering follower state: follower="Node at 3d248845-0412-bac6-239d-8fd9e990a7c9 [Follower]" leader-address= leader-id= -2024-07-05T13:29:57.054Z [WARN] server-3d248845-0412-bac6-239d-8fd9e990a7c9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:57.054Z [INFO] server-3d248845-0412-bac6-239d-8fd9e990a7c9: entering candidate state: node="Node at 3d248845-0412-bac6-239d-8fd9e990a7c9 [Candidate]" term=2 -2024-07-05T13:29:57.054Z [INFO] server-3d248845-0412-bac6-239d-8fd9e990a7c9: election won: term=2 tally=2 -2024-07-05T13:29:57.054Z [INFO] server-3d248845-0412-bac6-239d-8fd9e990a7c9: entering leader state: leader="Node at 3d248845-0412-bac6-239d-8fd9e990a7c9 [Leader]" -2024-07-05T13:29:57.054Z [INFO] server-3d248845-0412-bac6-239d-8fd9e990a7c9: added peer, starting replication: peer=server-2a7d3b39-116d-43a6-8c30-6669edfeb837 -2024-07-05T13:29:57.054Z [INFO] server-3d248845-0412-bac6-239d-8fd9e990a7c9: added peer, starting replication: peer=server-ee87242f-8340-62b7-d0ce-d0cb2c67894e -2024-07-05T13:29:57.054Z [INFO] server-3d248845-0412-bac6-239d-8fd9e990a7c9: pipelining replication: peer="{Voter server-ee87242f-8340-62b7-d0ce-d0cb2c67894e ee87242f-8340-62b7-d0ce-d0cb2c67894e}" -2024-07-05T13:29:57.054Z [INFO] server-3d248845-0412-bac6-239d-8fd9e990a7c9: pipelining replication: peer="{Voter server-2a7d3b39-116d-43a6-8c30-6669edfeb837 2a7d3b39-116d-43a6-8c30-6669edfeb837}" -2024-07-05T13:29:58.061Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:57.055043668 +0000 UTC m=+37.824954343 (1 nodes), 580.563153ms from start of poll, 581.390444ms from cluster start. Timeout at 2024-07-05 13:29:58.06119353 +0000 UTC m=+38.831104205, 1.006149862s after stability -2024-07-05T13:29:58.062Z [INFO] server-3d248845-0412-bac6-239d-8fd9e990a7c9: starting snapshot up to: index=12 -2024-07-05T13:29:58.062Z [INFO] creating new snapshot: path=/tmp/raft533346096/snapshots/2-12-1720186198062.tmp -2024-07-05T13:29:58.064Z [INFO] server-3d248845-0412-bac6-239d-8fd9e990a7c9: snapshot complete up to: index=12 -2024-07-05T13:29:58.067Z [INFO] server-bd926c84-9575-80e5-c745-92bf0e341049: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bd926c84-9575-80e5-c745-92bf0e341049 Address:bd926c84-9575-80e5-c745-92bf0e341049} {Suffrage:Voter ID:server-d6be4646-d9c6-57fc-fcd8-9fc17f003650 Address:d6be4646-d9c6-57fc-fcd8-9fc17f003650} {Suffrage:Voter ID:server-0de71213-7c93-1ce8-6d14-abaa92618359 Address:0de71213-7c93-1ce8-6d14-abaa92618359}]" -2024-07-05T13:29:58.067Z [INFO] server-d6be4646-d9c6-57fc-fcd8-9fc17f003650: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bd926c84-9575-80e5-c745-92bf0e341049 Address:bd926c84-9575-80e5-c745-92bf0e341049} {Suffrage:Voter ID:server-d6be4646-d9c6-57fc-fcd8-9fc17f003650 Address:d6be4646-d9c6-57fc-fcd8-9fc17f003650} {Suffrage:Voter ID:server-0de71213-7c93-1ce8-6d14-abaa92618359 Address:0de71213-7c93-1ce8-6d14-abaa92618359}]" -2024-07-05T13:29:58.067Z [INFO] server-0de71213-7c93-1ce8-6d14-abaa92618359: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bd926c84-9575-80e5-c745-92bf0e341049 Address:bd926c84-9575-80e5-c745-92bf0e341049} {Suffrage:Voter ID:server-d6be4646-d9c6-57fc-fcd8-9fc17f003650 Address:d6be4646-d9c6-57fc-fcd8-9fc17f003650} {Suffrage:Voter ID:server-0de71213-7c93-1ce8-6d14-abaa92618359 Address:0de71213-7c93-1ce8-6d14-abaa92618359}]" -2024-07-05T13:29:58.067Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:29:58.068Z [INFO] server-bd926c84-9575-80e5-c745-92bf0e341049: entering follower state: follower="Node at bd926c84-9575-80e5-c745-92bf0e341049 [Follower]" leader-address= leader-id= -2024-07-05T13:29:58.068Z [INFO] server-0de71213-7c93-1ce8-6d14-abaa92618359: entering follower state: follower="Node at 0de71213-7c93-1ce8-6d14-abaa92618359 [Follower]" leader-address= leader-id= -2024-07-05T13:29:58.068Z [INFO] server-d6be4646-d9c6-57fc-fcd8-9fc17f003650: entering follower state: follower="Node at d6be4646-d9c6-57fc-fcd8-9fc17f003650 [Follower]" leader-address= leader-id= -2024-07-05T13:29:58.590Z [WARN] server-0de71213-7c93-1ce8-6d14-abaa92618359: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:29:58.590Z [INFO] server-0de71213-7c93-1ce8-6d14-abaa92618359: entering candidate state: node="Node at 0de71213-7c93-1ce8-6d14-abaa92618359 [Candidate]" term=2 -2024-07-05T13:29:58.590Z [INFO] server-0de71213-7c93-1ce8-6d14-abaa92618359: election won: term=2 tally=2 -2024-07-05T13:29:58.590Z [INFO] server-0de71213-7c93-1ce8-6d14-abaa92618359: entering leader state: leader="Node at 0de71213-7c93-1ce8-6d14-abaa92618359 [Leader]" -2024-07-05T13:29:58.590Z [INFO] server-0de71213-7c93-1ce8-6d14-abaa92618359: added peer, starting replication: peer=server-bd926c84-9575-80e5-c745-92bf0e341049 -2024-07-05T13:29:58.590Z [INFO] server-0de71213-7c93-1ce8-6d14-abaa92618359: added peer, starting replication: peer=server-d6be4646-d9c6-57fc-fcd8-9fc17f003650 -2024-07-05T13:29:58.590Z [INFO] server-0de71213-7c93-1ce8-6d14-abaa92618359: pipelining replication: peer="{Voter server-d6be4646-d9c6-57fc-fcd8-9fc17f003650 d6be4646-d9c6-57fc-fcd8-9fc17f003650}" -2024-07-05T13:29:58.591Z [INFO] server-0de71213-7c93-1ce8-6d14-abaa92618359: pipelining replication: peer="{Voter server-bd926c84-9575-80e5-c745-92bf0e341049 bd926c84-9575-80e5-c745-92bf0e341049}" -2024-07-05T13:29:59.597Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:29:58.591158258 +0000 UTC m=+39.361068933 (1 nodes), 523.435477ms from start of poll, 524.095653ms from cluster start. Timeout at 2024-07-05 13:29:59.597437556 +0000 UTC m=+40.367348231, 1.006279298s after stability -2024-07-05T13:29:59.597Z [INFO] creating new snapshot: path=/tmp/raft3189538920/snapshots/2-14-1720186199597.tmp -2024-07-05T13:29:59.599Z [INFO] server-0de71213-7c93-1ce8-6d14-abaa92618359: copied to local snapshot: bytes=71 -2024-07-05T13:29:59.601Z [INFO] server-0de71213-7c93-1ce8-6d14-abaa92618359: snapshot restore progress: id=2-14-1720186199597 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:59.602Z [INFO] server-0de71213-7c93-1ce8-6d14-abaa92618359: removing all old logs from log store -2024-07-05T13:29:59.602Z [INFO] server-0de71213-7c93-1ce8-6d14-abaa92618359: compacting logs: from=1 to=2 -2024-07-05T13:29:59.602Z [INFO] server-0de71213-7c93-1ce8-6d14-abaa92618359: restored user snapshot: index=14 -2024-07-05T13:29:59.603Z [ERROR] server-0de71213-7c93-1ce8-6d14-abaa92618359: failed to get log: index=2 error="log not found" -2024-07-05T13:29:59.603Z [INFO] server-0de71213-7c93-1ce8-6d14-abaa92618359: aborting pipeline replication: peer="{Voter server-bd926c84-9575-80e5-c745-92bf0e341049 bd926c84-9575-80e5-c745-92bf0e341049}" -2024-07-05T13:29:59.606Z [ERROR] server-0de71213-7c93-1ce8-6d14-abaa92618359: failed to get log: index=2 error="log not found" -2024-07-05T13:29:59.606Z [INFO] server-0de71213-7c93-1ce8-6d14-abaa92618359: aborting pipeline replication: peer="{Voter server-d6be4646-d9c6-57fc-fcd8-9fc17f003650 d6be4646-d9c6-57fc-fcd8-9fc17f003650}" -2024-07-05T13:29:59.609Z [ERROR] server-0de71213-7c93-1ce8-6d14-abaa92618359: failed to get log: index=2 error="log not found" -2024-07-05T13:29:59.610Z [INFO] creating new snapshot: path=/tmp/raft4110600379/snapshots/2-14-1720186199610.tmp -2024-07-05T13:29:59.611Z [INFO] server-bd926c84-9575-80e5-c745-92bf0e341049: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:59.611Z [INFO] server-bd926c84-9575-80e5-c745-92bf0e341049: copied to local snapshot: bytes=71 -2024-07-05T13:29:59.612Z [INFO] server-bd926c84-9575-80e5-c745-92bf0e341049: snapshot restore progress: id=2-14-1720186199610 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:59.612Z [INFO] server-bd926c84-9575-80e5-c745-92bf0e341049: removing all old logs from log store -2024-07-05T13:29:59.612Z [INFO] server-bd926c84-9575-80e5-c745-92bf0e341049: compacting logs: from=1 to=2 -2024-07-05T13:29:59.612Z [INFO] server-bd926c84-9575-80e5-c745-92bf0e341049: Installed remote snapshot -2024-07-05T13:29:59.612Z [INFO] server-0de71213-7c93-1ce8-6d14-abaa92618359: pipelining replication: peer="{Voter server-bd926c84-9575-80e5-c745-92bf0e341049 bd926c84-9575-80e5-c745-92bf0e341049}" -2024-07-05T13:29:59.612Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:29:59.616Z [ERROR] server-0de71213-7c93-1ce8-6d14-abaa92618359: failed to get log: index=2 error="log not found" -2024-07-05T13:29:59.617Z [INFO] creating new snapshot: path=/tmp/raft3306051932/snapshots/2-14-1720186199617.tmp -2024-07-05T13:29:59.617Z [INFO] server-d6be4646-d9c6-57fc-fcd8-9fc17f003650: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:59.618Z [INFO] server-d6be4646-d9c6-57fc-fcd8-9fc17f003650: copied to local snapshot: bytes=71 -2024-07-05T13:29:59.618Z [INFO] server-d6be4646-d9c6-57fc-fcd8-9fc17f003650: snapshot restore progress: id=2-14-1720186199617 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:29:59.618Z [INFO] server-d6be4646-d9c6-57fc-fcd8-9fc17f003650: removing all old logs from log store -2024-07-05T13:29:59.618Z [INFO] server-d6be4646-d9c6-57fc-fcd8-9fc17f003650: compacting logs: from=1 to=2 -2024-07-05T13:29:59.618Z [INFO] server-d6be4646-d9c6-57fc-fcd8-9fc17f003650: Installed remote snapshot -2024-07-05T13:29:59.618Z [INFO] server-0de71213-7c93-1ce8-6d14-abaa92618359: pipelining replication: peer="{Voter server-d6be4646-d9c6-57fc-fcd8-9fc17f003650 d6be4646-d9c6-57fc-fcd8-9fc17f003650}" -2024-07-05T13:30:00.618Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:29:59.612735819 +0000 UTC m=+40.382646494 (2 nodes), 1.56µs from start of poll, 1.545673214s from cluster start. Timeout at 2024-07-05 13:30:00.618205906 +0000 UTC m=+41.388116581, 1.005470087s after stability -2024-07-05T13:30:00.630Z [INFO] server-3d248845-0412-bac6-239d-8fd9e990a7c9: aborting pipeline replication: peer="{Voter server-ee87242f-8340-62b7-d0ce-d0cb2c67894e ee87242f-8340-62b7-d0ce-d0cb2c67894e}" -2024-07-05T13:30:00.630Z [INFO] server-3d248845-0412-bac6-239d-8fd9e990a7c9: aborting pipeline replication: peer="{Voter server-2a7d3b39-116d-43a6-8c30-6669edfeb837 2a7d3b39-116d-43a6-8c30-6669edfeb837}" +2024-07-05T13:36:12.581Z [INFO] server-ccba7c8c-7f0e-d968-2e63-561b8e560446: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccba7c8c-7f0e-d968-2e63-561b8e560446 Address:ccba7c8c-7f0e-d968-2e63-561b8e560446} {Suffrage:Voter ID:server-429c6931-105a-80ae-df0d-2022ef045fb2 Address:429c6931-105a-80ae-df0d-2022ef045fb2} {Suffrage:Voter ID:server-85aeef15-54af-d392-5ac3-45ce678ac2eb Address:85aeef15-54af-d392-5ac3-45ce678ac2eb}]" +2024-07-05T13:36:12.581Z [INFO] server-ccba7c8c-7f0e-d968-2e63-561b8e560446: entering follower state: follower="Node at ccba7c8c-7f0e-d968-2e63-561b8e560446 [Follower]" leader-address= leader-id= +2024-07-05T13:36:12.582Z [INFO] server-429c6931-105a-80ae-df0d-2022ef045fb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccba7c8c-7f0e-d968-2e63-561b8e560446 Address:ccba7c8c-7f0e-d968-2e63-561b8e560446} {Suffrage:Voter ID:server-429c6931-105a-80ae-df0d-2022ef045fb2 Address:429c6931-105a-80ae-df0d-2022ef045fb2} {Suffrage:Voter ID:server-85aeef15-54af-d392-5ac3-45ce678ac2eb Address:85aeef15-54af-d392-5ac3-45ce678ac2eb}]" +2024-07-05T13:36:12.582Z [INFO] server-429c6931-105a-80ae-df0d-2022ef045fb2: entering follower state: follower="Node at 429c6931-105a-80ae-df0d-2022ef045fb2 [Follower]" leader-address= leader-id= +2024-07-05T13:36:12.583Z [INFO] server-85aeef15-54af-d392-5ac3-45ce678ac2eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccba7c8c-7f0e-d968-2e63-561b8e560446 Address:ccba7c8c-7f0e-d968-2e63-561b8e560446} {Suffrage:Voter ID:server-429c6931-105a-80ae-df0d-2022ef045fb2 Address:429c6931-105a-80ae-df0d-2022ef045fb2} {Suffrage:Voter ID:server-85aeef15-54af-d392-5ac3-45ce678ac2eb Address:85aeef15-54af-d392-5ac3-45ce678ac2eb}]" +2024-07-05T13:36:12.583Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:12.583Z [INFO] server-85aeef15-54af-d392-5ac3-45ce678ac2eb: entering follower state: follower="Node at 85aeef15-54af-d392-5ac3-45ce678ac2eb [Follower]" leader-address= leader-id= +2024-07-05T13:36:13.089Z [WARN] server-ccba7c8c-7f0e-d968-2e63-561b8e560446: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:13.089Z [INFO] server-ccba7c8c-7f0e-d968-2e63-561b8e560446: entering candidate state: node="Node at ccba7c8c-7f0e-d968-2e63-561b8e560446 [Candidate]" term=2 +2024-07-05T13:36:13.090Z [INFO] server-ccba7c8c-7f0e-d968-2e63-561b8e560446: election won: term=2 tally=2 +2024-07-05T13:36:13.090Z [INFO] server-ccba7c8c-7f0e-d968-2e63-561b8e560446: entering leader state: leader="Node at ccba7c8c-7f0e-d968-2e63-561b8e560446 [Leader]" +2024-07-05T13:36:13.090Z [INFO] server-ccba7c8c-7f0e-d968-2e63-561b8e560446: added peer, starting replication: peer=server-429c6931-105a-80ae-df0d-2022ef045fb2 +2024-07-05T13:36:13.091Z [INFO] server-ccba7c8c-7f0e-d968-2e63-561b8e560446: added peer, starting replication: peer=server-85aeef15-54af-d392-5ac3-45ce678ac2eb +2024-07-05T13:36:13.092Z [INFO] server-ccba7c8c-7f0e-d968-2e63-561b8e560446: pipelining replication: peer="{Voter server-85aeef15-54af-d392-5ac3-45ce678ac2eb 85aeef15-54af-d392-5ac3-45ce678ac2eb}" +2024-07-05T13:36:13.092Z [INFO] server-ccba7c8c-7f0e-d968-2e63-561b8e560446: pipelining replication: peer="{Voter server-429c6931-105a-80ae-df0d-2022ef045fb2 429c6931-105a-80ae-df0d-2022ef045fb2}" +2024-07-05T13:36:14.097Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:13.09185573 +0000 UTC m=+39.087777182 (1 nodes), 508.222218ms from start of poll, 511.463159ms from cluster start. Timeout at 2024-07-05 13:36:14.097608152 +0000 UTC m=+40.093529603, 1.005752421s after stability +2024-07-05T13:36:14.102Z [INFO] server-ccba7c8c-7f0e-d968-2e63-561b8e560446: starting snapshot up to: index=12 +2024-07-05T13:36:14.103Z [INFO] creating new snapshot: path=/tmp/raft3496971235/snapshots/2-12-1720186574103.tmp +2024-07-05T13:36:14.110Z [INFO] server-ccba7c8c-7f0e-d968-2e63-561b8e560446: snapshot complete up to: index=12 +2024-07-05T13:36:14.124Z [INFO] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d81219f8-734a-b5cf-d519-6b3b234af2a5 Address:d81219f8-734a-b5cf-d519-6b3b234af2a5} {Suffrage:Voter ID:server-7a7f2959-0dcd-9779-574e-d2d1f623fc73 Address:7a7f2959-0dcd-9779-574e-d2d1f623fc73} {Suffrage:Voter ID:server-f82898c0-3495-5466-c320-3e7cbdb788ca Address:f82898c0-3495-5466-c320-3e7cbdb788ca}]" +2024-07-05T13:36:14.125Z [INFO] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: entering follower state: follower="Node at d81219f8-734a-b5cf-d519-6b3b234af2a5 [Follower]" leader-address= leader-id= +2024-07-05T13:36:14.125Z [INFO] server-7a7f2959-0dcd-9779-574e-d2d1f623fc73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d81219f8-734a-b5cf-d519-6b3b234af2a5 Address:d81219f8-734a-b5cf-d519-6b3b234af2a5} {Suffrage:Voter ID:server-7a7f2959-0dcd-9779-574e-d2d1f623fc73 Address:7a7f2959-0dcd-9779-574e-d2d1f623fc73} {Suffrage:Voter ID:server-f82898c0-3495-5466-c320-3e7cbdb788ca Address:f82898c0-3495-5466-c320-3e7cbdb788ca}]" +2024-07-05T13:36:14.126Z [INFO] server-7a7f2959-0dcd-9779-574e-d2d1f623fc73: entering follower state: follower="Node at 7a7f2959-0dcd-9779-574e-d2d1f623fc73 [Follower]" leader-address= leader-id= +2024-07-05T13:36:14.126Z [INFO] server-f82898c0-3495-5466-c320-3e7cbdb788ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d81219f8-734a-b5cf-d519-6b3b234af2a5 Address:d81219f8-734a-b5cf-d519-6b3b234af2a5} {Suffrage:Voter ID:server-7a7f2959-0dcd-9779-574e-d2d1f623fc73 Address:7a7f2959-0dcd-9779-574e-d2d1f623fc73} {Suffrage:Voter ID:server-f82898c0-3495-5466-c320-3e7cbdb788ca Address:f82898c0-3495-5466-c320-3e7cbdb788ca}]" +2024-07-05T13:36:14.127Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:14.127Z [INFO] server-f82898c0-3495-5466-c320-3e7cbdb788ca: entering follower state: follower="Node at f82898c0-3495-5466-c320-3e7cbdb788ca [Follower]" leader-address= leader-id= +2024-07-05T13:36:15.081Z [WARN] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:15.082Z [INFO] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: entering candidate state: node="Node at d81219f8-734a-b5cf-d519-6b3b234af2a5 [Candidate]" term=2 +2024-07-05T13:36:15.082Z [INFO] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: election won: term=2 tally=2 +2024-07-05T13:36:15.082Z [INFO] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: entering leader state: leader="Node at d81219f8-734a-b5cf-d519-6b3b234af2a5 [Leader]" +2024-07-05T13:36:15.083Z [INFO] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: added peer, starting replication: peer=server-7a7f2959-0dcd-9779-574e-d2d1f623fc73 +2024-07-05T13:36:15.083Z [INFO] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: added peer, starting replication: peer=server-f82898c0-3495-5466-c320-3e7cbdb788ca +2024-07-05T13:36:15.084Z [INFO] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: pipelining replication: peer="{Voter server-f82898c0-3495-5466-c320-3e7cbdb788ca f82898c0-3495-5466-c320-3e7cbdb788ca}" +2024-07-05T13:36:15.084Z [INFO] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: pipelining replication: peer="{Voter server-7a7f2959-0dcd-9779-574e-d2d1f623fc73 7a7f2959-0dcd-9779-574e-d2d1f623fc73}" +2024-07-05T13:36:16.088Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:15.082992987 +0000 UTC m=+41.078914439 (1 nodes), 955.681462ms from start of poll, 958.773655ms from cluster start. Timeout at 2024-07-05 13:36:16.088232377 +0000 UTC m=+42.084153828, 1.005239389s after stability +2024-07-05T13:36:16.090Z [INFO] creating new snapshot: path=/tmp/raft2152073257/snapshots/2-14-1720186576090.tmp +2024-07-05T13:36:16.094Z [INFO] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: copied to local snapshot: bytes=71 +2024-07-05T13:36:16.096Z [INFO] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: snapshot restore progress: id=2-14-1720186576090 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:16.096Z [INFO] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: removing all old logs from log store +2024-07-05T13:36:16.096Z [INFO] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: compacting logs: from=1 to=2 +2024-07-05T13:36:16.096Z [INFO] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: restored user snapshot: index=14 +2024-07-05T13:36:16.097Z [ERROR] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: failed to get log: index=2 error="log not found" +2024-07-05T13:36:16.097Z [ERROR] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: failed to get log: index=2 error="log not found" +2024-07-05T13:36:16.097Z [INFO] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: aborting pipeline replication: peer="{Voter server-f82898c0-3495-5466-c320-3e7cbdb788ca f82898c0-3495-5466-c320-3e7cbdb788ca}" +2024-07-05T13:36:16.097Z [INFO] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: aborting pipeline replication: peer="{Voter server-7a7f2959-0dcd-9779-574e-d2d1f623fc73 7a7f2959-0dcd-9779-574e-d2d1f623fc73}" +2024-07-05T13:36:16.103Z [ERROR] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: failed to get log: index=2 error="log not found" +2024-07-05T13:36:16.105Z [INFO] creating new snapshot: path=/tmp/raft1897450/snapshots/2-14-1720186576105.tmp +2024-07-05T13:36:16.107Z [ERROR] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: failed to get log: index=2 error="log not found" +2024-07-05T13:36:16.107Z [INFO] server-f82898c0-3495-5466-c320-3e7cbdb788ca: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:16.110Z [INFO] server-f82898c0-3495-5466-c320-3e7cbdb788ca: copied to local snapshot: bytes=71 +2024-07-05T13:36:16.110Z [INFO] creating new snapshot: path=/tmp/raft1744234948/snapshots/2-14-1720186576110.tmp +2024-07-05T13:36:16.111Z [INFO] server-f82898c0-3495-5466-c320-3e7cbdb788ca: snapshot restore progress: id=2-14-1720186576105 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:16.112Z [INFO] server-f82898c0-3495-5466-c320-3e7cbdb788ca: removing all old logs from log store +2024-07-05T13:36:16.112Z [INFO] server-f82898c0-3495-5466-c320-3e7cbdb788ca: compacting logs: from=1 to=2 +2024-07-05T13:36:16.112Z [INFO] server-f82898c0-3495-5466-c320-3e7cbdb788ca: Installed remote snapshot +2024-07-05T13:36:16.112Z [INFO] server-7a7f2959-0dcd-9779-574e-d2d1f623fc73: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:16.113Z [INFO] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: pipelining replication: peer="{Voter server-f82898c0-3495-5466-c320-3e7cbdb788ca f82898c0-3495-5466-c320-3e7cbdb788ca}" +2024-07-05T13:36:16.113Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:36:16.115Z [INFO] server-7a7f2959-0dcd-9779-574e-d2d1f623fc73: copied to local snapshot: bytes=71 +2024-07-05T13:36:16.116Z [INFO] server-7a7f2959-0dcd-9779-574e-d2d1f623fc73: snapshot restore progress: id=2-14-1720186576110 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:16.117Z [INFO] server-7a7f2959-0dcd-9779-574e-d2d1f623fc73: removing all old logs from log store +2024-07-05T13:36:16.117Z [INFO] server-7a7f2959-0dcd-9779-574e-d2d1f623fc73: compacting logs: from=1 to=2 +2024-07-05T13:36:16.117Z [INFO] server-7a7f2959-0dcd-9779-574e-d2d1f623fc73: Installed remote snapshot +2024-07-05T13:36:16.118Z [INFO] server-d81219f8-734a-b5cf-d519-6b3b234af2a5: pipelining replication: peer="{Voter server-7a7f2959-0dcd-9779-574e-d2d1f623fc73 7a7f2959-0dcd-9779-574e-d2d1f623fc73}" +2024-07-05T13:36:17.119Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:36:16.113746911 +0000 UTC m=+42.109668362 (2 nodes), 8.458µs from start of poll, 1.989527578s from cluster start. Timeout at 2024-07-05 13:36:17.11927067 +0000 UTC m=+43.115192122, 1.00552376s after stability +2024-07-05T13:36:17.135Z [INFO] server-ccba7c8c-7f0e-d968-2e63-561b8e560446: aborting pipeline replication: peer="{Voter server-85aeef15-54af-d392-5ac3-45ce678ac2eb 85aeef15-54af-d392-5ac3-45ce678ac2eb}" +2024-07-05T13:36:17.135Z [INFO] server-ccba7c8c-7f0e-d968-2e63-561b8e560446: aborting pipeline replication: peer="{Voter server-429c6931-105a-80ae-df0d-2022ef045fb2 429c6931-105a-80ae-df0d-2022ef045fb2}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false -2024-07-05T13:30:00.633Z [INFO] server-a2b6c2a1-17b5-900b-c149-c07d97d798a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a2b6c2a1-17b5-900b-c149-c07d97d798a2 Address:a2b6c2a1-17b5-900b-c149-c07d97d798a2} {Suffrage:Voter ID:server-2ac78561-a1e7-16a6-0d13-79367ada1a69 Address:2ac78561-a1e7-16a6-0d13-79367ada1a69} {Suffrage:Voter ID:server-765f258f-ec27-f116-e721-62a9a95a358d Address:765f258f-ec27-f116-e721-62a9a95a358d}]" -2024-07-05T13:30:00.633Z [INFO] server-2ac78561-a1e7-16a6-0d13-79367ada1a69: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a2b6c2a1-17b5-900b-c149-c07d97d798a2 Address:a2b6c2a1-17b5-900b-c149-c07d97d798a2} {Suffrage:Voter ID:server-2ac78561-a1e7-16a6-0d13-79367ada1a69 Address:2ac78561-a1e7-16a6-0d13-79367ada1a69} {Suffrage:Voter ID:server-765f258f-ec27-f116-e721-62a9a95a358d Address:765f258f-ec27-f116-e721-62a9a95a358d}]" -2024-07-05T13:30:00.633Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a2b6c2a1-17b5-900b-c149-c07d97d798a2 Address:a2b6c2a1-17b5-900b-c149-c07d97d798a2} {Suffrage:Voter ID:server-2ac78561-a1e7-16a6-0d13-79367ada1a69 Address:2ac78561-a1e7-16a6-0d13-79367ada1a69} {Suffrage:Voter ID:server-765f258f-ec27-f116-e721-62a9a95a358d Address:765f258f-ec27-f116-e721-62a9a95a358d}]" -2024-07-05T13:30:00.633Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:00.633Z [INFO] server-a2b6c2a1-17b5-900b-c149-c07d97d798a2: entering follower state: follower="Node at a2b6c2a1-17b5-900b-c149-c07d97d798a2 [Follower]" leader-address= leader-id= -2024-07-05T13:30:00.633Z [INFO] server-2ac78561-a1e7-16a6-0d13-79367ada1a69: entering follower state: follower="Node at 2ac78561-a1e7-16a6-0d13-79367ada1a69 [Follower]" leader-address= leader-id= -2024-07-05T13:30:00.634Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: entering follower state: follower="Node at 765f258f-ec27-f116-e721-62a9a95a358d [Follower]" leader-address= leader-id= -2024-07-05T13:30:01.241Z [WARN] server-765f258f-ec27-f116-e721-62a9a95a358d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:01.241Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: entering candidate state: node="Node at 765f258f-ec27-f116-e721-62a9a95a358d [Candidate]" term=2 -2024-07-05T13:30:01.241Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: election won: term=2 tally=2 -2024-07-05T13:30:01.241Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: entering leader state: leader="Node at 765f258f-ec27-f116-e721-62a9a95a358d [Leader]" -2024-07-05T13:30:01.241Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: added peer, starting replication: peer=server-a2b6c2a1-17b5-900b-c149-c07d97d798a2 -2024-07-05T13:30:01.241Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: added peer, starting replication: peer=server-2ac78561-a1e7-16a6-0d13-79367ada1a69 -2024-07-05T13:30:01.242Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: pipelining replication: peer="{Voter server-a2b6c2a1-17b5-900b-c149-c07d97d798a2 a2b6c2a1-17b5-900b-c149-c07d97d798a2}" -2024-07-05T13:30:01.242Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: pipelining replication: peer="{Voter server-2ac78561-a1e7-16a6-0d13-79367ada1a69 2ac78561-a1e7-16a6-0d13-79367ada1a69}" -2024-07-05T13:30:02.248Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:01.242634879 +0000 UTC m=+42.012545554 (1 nodes), 608.771203ms from start of poll, 609.498617ms from cluster start. Timeout at 2024-07-05 13:30:02.24797545 +0000 UTC m=+43.017886125, 1.005340571s after stability -2024-07-05T13:30:02.249Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: starting snapshot up to: index=12 -2024-07-05T13:30:02.249Z [INFO] creating new snapshot: path=/tmp/raft764571943/snapshots/2-12-1720186202249.tmp -2024-07-05T13:30:02.250Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: snapshot complete up to: index=12 -2024-07-05T13:30:02.253Z [INFO] creating new snapshot: path=/tmp/raft764571943/snapshots/2-23-1720186202253.tmp -2024-07-05T13:30:02.254Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: copied to local snapshot: bytes=71 -2024-07-05T13:30:02.262Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: snapshot restore progress: id=2-23-1720186202253 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:02.262Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: restored user snapshot: index=23 -2024-07-05T13:30:02.262Z [ERROR] server-765f258f-ec27-f116-e721-62a9a95a358d: failed to get log: index=23 error="log not found" -2024-07-05T13:30:02.262Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: aborting pipeline replication: peer="{Voter server-2ac78561-a1e7-16a6-0d13-79367ada1a69 2ac78561-a1e7-16a6-0d13-79367ada1a69}" -2024-07-05T13:30:02.262Z [ERROR] server-765f258f-ec27-f116-e721-62a9a95a358d: failed to get log: index=23 error="log not found" -2024-07-05T13:30:02.262Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: aborting pipeline replication: peer="{Voter server-a2b6c2a1-17b5-900b-c149-c07d97d798a2 a2b6c2a1-17b5-900b-c149-c07d97d798a2}" -2024-07-05T13:30:02.268Z [ERROR] server-765f258f-ec27-f116-e721-62a9a95a358d: failed to get log: index=23 error="log not found" -2024-07-05T13:30:02.269Z [INFO] creating new snapshot: path=/tmp/raft1266999642/snapshots/2-23-1720186202269.tmp -2024-07-05T13:30:02.269Z [INFO] server-a2b6c2a1-17b5-900b-c149-c07d97d798a2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:02.270Z [INFO] server-a2b6c2a1-17b5-900b-c149-c07d97d798a2: copied to local snapshot: bytes=71 -2024-07-05T13:30:02.271Z [INFO] server-a2b6c2a1-17b5-900b-c149-c07d97d798a2: snapshot restore progress: id=2-23-1720186202269 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:02.271Z [INFO] server-a2b6c2a1-17b5-900b-c149-c07d97d798a2: Installed remote snapshot -2024-07-05T13:30:02.271Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: pipelining replication: peer="{Voter server-a2b6c2a1-17b5-900b-c149-c07d97d798a2 a2b6c2a1-17b5-900b-c149-c07d97d798a2}" -2024-07-05T13:30:02.271Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:02.272Z [ERROR] server-765f258f-ec27-f116-e721-62a9a95a358d: failed to get log: index=23 error="log not found" -2024-07-05T13:30:02.273Z [INFO] creating new snapshot: path=/tmp/raft1351921603/snapshots/2-23-1720186202273.tmp -2024-07-05T13:30:02.273Z [INFO] server-2ac78561-a1e7-16a6-0d13-79367ada1a69: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:02.274Z [INFO] server-2ac78561-a1e7-16a6-0d13-79367ada1a69: copied to local snapshot: bytes=71 -2024-07-05T13:30:02.274Z [INFO] server-2ac78561-a1e7-16a6-0d13-79367ada1a69: snapshot restore progress: id=2-23-1720186202273 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:02.274Z [INFO] server-2ac78561-a1e7-16a6-0d13-79367ada1a69: Installed remote snapshot -2024-07-05T13:30:02.275Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: pipelining replication: peer="{Voter server-2ac78561-a1e7-16a6-0d13-79367ada1a69 2ac78561-a1e7-16a6-0d13-79367ada1a69}" -2024-07-05T13:30:03.277Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:02.271755776 +0000 UTC m=+43.041666451 (2 nodes), 2.24µs from start of poll, 1.638619514s from cluster start. Timeout at 2024-07-05 13:30:03.277098466 +0000 UTC m=+44.047009181, 1.00534273s after stability -2024-07-05T13:30:03.289Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: aborting pipeline replication: peer="{Voter server-2ac78561-a1e7-16a6-0d13-79367ada1a69 2ac78561-a1e7-16a6-0d13-79367ada1a69}" -2024-07-05T13:30:03.289Z [INFO] server-765f258f-ec27-f116-e721-62a9a95a358d: aborting pipeline replication: peer="{Voter server-a2b6c2a1-17b5-900b-c149-c07d97d798a2 a2b6c2a1-17b5-900b-c149-c07d97d798a2}" +2024-07-05T13:36:17.154Z [INFO] server-16c21642-aa3c-a9c1-65c6-21a1dd087319: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c21642-aa3c-a9c1-65c6-21a1dd087319 Address:16c21642-aa3c-a9c1-65c6-21a1dd087319} {Suffrage:Voter ID:server-572b0f85-285d-b9c2-6bd8-d5768be917da Address:572b0f85-285d-b9c2-6bd8-d5768be917da} {Suffrage:Voter ID:server-3ca0ac97-3834-4105-be1a-34a8ca3011a1 Address:3ca0ac97-3834-4105-be1a-34a8ca3011a1}]" +2024-07-05T13:36:17.154Z [INFO] server-16c21642-aa3c-a9c1-65c6-21a1dd087319: entering follower state: follower="Node at 16c21642-aa3c-a9c1-65c6-21a1dd087319 [Follower]" leader-address= leader-id= +2024-07-05T13:36:17.155Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c21642-aa3c-a9c1-65c6-21a1dd087319 Address:16c21642-aa3c-a9c1-65c6-21a1dd087319} {Suffrage:Voter ID:server-572b0f85-285d-b9c2-6bd8-d5768be917da Address:572b0f85-285d-b9c2-6bd8-d5768be917da} {Suffrage:Voter ID:server-3ca0ac97-3834-4105-be1a-34a8ca3011a1 Address:3ca0ac97-3834-4105-be1a-34a8ca3011a1}]" +2024-07-05T13:36:17.155Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: entering follower state: follower="Node at 572b0f85-285d-b9c2-6bd8-d5768be917da [Follower]" leader-address= leader-id= +2024-07-05T13:36:17.156Z [INFO] server-3ca0ac97-3834-4105-be1a-34a8ca3011a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c21642-aa3c-a9c1-65c6-21a1dd087319 Address:16c21642-aa3c-a9c1-65c6-21a1dd087319} {Suffrage:Voter ID:server-572b0f85-285d-b9c2-6bd8-d5768be917da Address:572b0f85-285d-b9c2-6bd8-d5768be917da} {Suffrage:Voter ID:server-3ca0ac97-3834-4105-be1a-34a8ca3011a1 Address:3ca0ac97-3834-4105-be1a-34a8ca3011a1}]" +2024-07-05T13:36:17.156Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:17.156Z [INFO] server-3ca0ac97-3834-4105-be1a-34a8ca3011a1: entering follower state: follower="Node at 3ca0ac97-3834-4105-be1a-34a8ca3011a1 [Follower]" leader-address= leader-id= +2024-07-05T13:36:17.974Z [WARN] server-572b0f85-285d-b9c2-6bd8-d5768be917da: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:17.974Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: entering candidate state: node="Node at 572b0f85-285d-b9c2-6bd8-d5768be917da [Candidate]" term=2 +2024-07-05T13:36:17.975Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: election won: term=2 tally=2 +2024-07-05T13:36:17.975Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: entering leader state: leader="Node at 572b0f85-285d-b9c2-6bd8-d5768be917da [Leader]" +2024-07-05T13:36:17.975Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: added peer, starting replication: peer=server-16c21642-aa3c-a9c1-65c6-21a1dd087319 +2024-07-05T13:36:17.975Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: added peer, starting replication: peer=server-3ca0ac97-3834-4105-be1a-34a8ca3011a1 +2024-07-05T13:36:17.976Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: pipelining replication: peer="{Voter server-16c21642-aa3c-a9c1-65c6-21a1dd087319 16c21642-aa3c-a9c1-65c6-21a1dd087319}" +2024-07-05T13:36:17.976Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: pipelining replication: peer="{Voter server-3ca0ac97-3834-4105-be1a-34a8ca3011a1 3ca0ac97-3834-4105-be1a-34a8ca3011a1}" +2024-07-05T13:36:18.980Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:17.975704444 +0000 UTC m=+43.971626187 (1 nodes), 818.876543ms from start of poll, 821.833697ms from cluster start. Timeout at 2024-07-05 13:36:18.980740254 +0000 UTC m=+44.976661706, 1.005035519s after stability +2024-07-05T13:36:18.985Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: starting snapshot up to: index=12 +2024-07-05T13:36:18.986Z [INFO] creating new snapshot: path=/tmp/raft4107319111/snapshots/2-12-1720186578986.tmp +2024-07-05T13:36:18.990Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: snapshot complete up to: index=12 +2024-07-05T13:36:18.996Z [INFO] creating new snapshot: path=/tmp/raft4107319111/snapshots/2-23-1720186578996.tmp +2024-07-05T13:36:19.000Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: copied to local snapshot: bytes=71 +2024-07-05T13:36:19.002Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: snapshot restore progress: id=2-23-1720186578996 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:19.002Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: restored user snapshot: index=23 +2024-07-05T13:36:19.002Z [ERROR] server-572b0f85-285d-b9c2-6bd8-d5768be917da: failed to get log: index=23 error="log not found" +2024-07-05T13:36:19.002Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: aborting pipeline replication: peer="{Voter server-16c21642-aa3c-a9c1-65c6-21a1dd087319 16c21642-aa3c-a9c1-65c6-21a1dd087319}" +2024-07-05T13:36:19.002Z [ERROR] server-572b0f85-285d-b9c2-6bd8-d5768be917da: failed to get log: index=23 error="log not found" +2024-07-05T13:36:19.003Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: aborting pipeline replication: peer="{Voter server-3ca0ac97-3834-4105-be1a-34a8ca3011a1 3ca0ac97-3834-4105-be1a-34a8ca3011a1}" +2024-07-05T13:36:19.012Z [ERROR] server-572b0f85-285d-b9c2-6bd8-d5768be917da: failed to get log: index=23 error="log not found" +2024-07-05T13:36:19.013Z [ERROR] server-572b0f85-285d-b9c2-6bd8-d5768be917da: failed to get log: index=23 error="log not found" +2024-07-05T13:36:19.015Z [INFO] creating new snapshot: path=/tmp/raft1660139550/snapshots/2-23-1720186579015.tmp +2024-07-05T13:36:19.016Z [INFO] creating new snapshot: path=/tmp/raft578237316/snapshots/2-23-1720186579016.tmp +2024-07-05T13:36:19.016Z [INFO] server-16c21642-aa3c-a9c1-65c6-21a1dd087319: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:19.017Z [INFO] server-3ca0ac97-3834-4105-be1a-34a8ca3011a1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:19.019Z [INFO] server-16c21642-aa3c-a9c1-65c6-21a1dd087319: copied to local snapshot: bytes=71 +2024-07-05T13:36:19.020Z [INFO] server-3ca0ac97-3834-4105-be1a-34a8ca3011a1: copied to local snapshot: bytes=71 +2024-07-05T13:36:19.026Z [INFO] server-3ca0ac97-3834-4105-be1a-34a8ca3011a1: snapshot restore progress: id=2-23-1720186579016 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:19.028Z [INFO] server-16c21642-aa3c-a9c1-65c6-21a1dd087319: snapshot restore progress: id=2-23-1720186579015 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:19.029Z [INFO] server-16c21642-aa3c-a9c1-65c6-21a1dd087319: Installed remote snapshot +2024-07-05T13:36:19.031Z [INFO] server-3ca0ac97-3834-4105-be1a-34a8ca3011a1: Installed remote snapshot +2024-07-05T13:36:19.032Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: pipelining replication: peer="{Voter server-16c21642-aa3c-a9c1-65c6-21a1dd087319 16c21642-aa3c-a9c1-65c6-21a1dd087319}" +2024-07-05T13:36:19.032Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:36:19.033Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: pipelining replication: peer="{Voter server-3ca0ac97-3834-4105-be1a-34a8ca3011a1 3ca0ac97-3834-4105-be1a-34a8ca3011a1}" +2024-07-05T13:36:20.039Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:36:19.032897468 +0000 UTC m=+45.028818920 (2 nodes), 8.167µs from start of poll, 1.87902643s from cluster start. Timeout at 2024-07-05 13:36:20.038905677 +0000 UTC m=+46.034827129, 1.006008209s after stability +2024-07-05T13:36:20.057Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: aborting pipeline replication: peer="{Voter server-16c21642-aa3c-a9c1-65c6-21a1dd087319 16c21642-aa3c-a9c1-65c6-21a1dd087319}" +2024-07-05T13:36:20.057Z [INFO] server-572b0f85-285d-b9c2-6bd8-d5768be917da: aborting pipeline replication: peer="{Voter server-3ca0ac97-3834-4105-be1a-34a8ca3011a1 3ca0ac97-3834-4105-be1a-34a8ca3011a1}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false -2024-07-05T13:30:03.293Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc Address:930cee5f-afaa-ee8c-62e1-4c981d8a51bc} {Suffrage:Voter ID:server-0701f8bc-7092-53a8-ffb6-e79314e819c8 Address:0701f8bc-7092-53a8-ffb6-e79314e819c8} {Suffrage:Voter ID:server-df302265-7ca7-da7c-d545-835cf35c578b Address:df302265-7ca7-da7c-d545-835cf35c578b}]" -2024-07-05T13:30:03.293Z [INFO] server-0701f8bc-7092-53a8-ffb6-e79314e819c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc Address:930cee5f-afaa-ee8c-62e1-4c981d8a51bc} {Suffrage:Voter ID:server-0701f8bc-7092-53a8-ffb6-e79314e819c8 Address:0701f8bc-7092-53a8-ffb6-e79314e819c8} {Suffrage:Voter ID:server-df302265-7ca7-da7c-d545-835cf35c578b Address:df302265-7ca7-da7c-d545-835cf35c578b}]" -2024-07-05T13:30:03.293Z [INFO] server-df302265-7ca7-da7c-d545-835cf35c578b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc Address:930cee5f-afaa-ee8c-62e1-4c981d8a51bc} {Suffrage:Voter ID:server-0701f8bc-7092-53a8-ffb6-e79314e819c8 Address:0701f8bc-7092-53a8-ffb6-e79314e819c8} {Suffrage:Voter ID:server-df302265-7ca7-da7c-d545-835cf35c578b Address:df302265-7ca7-da7c-d545-835cf35c578b}]" -2024-07-05T13:30:03.293Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:03.293Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: entering follower state: follower="Node at 930cee5f-afaa-ee8c-62e1-4c981d8a51bc [Follower]" leader-address= leader-id= -2024-07-05T13:30:03.293Z [INFO] server-0701f8bc-7092-53a8-ffb6-e79314e819c8: entering follower state: follower="Node at 0701f8bc-7092-53a8-ffb6-e79314e819c8 [Follower]" leader-address= leader-id= -2024-07-05T13:30:03.293Z [INFO] server-df302265-7ca7-da7c-d545-835cf35c578b: entering follower state: follower="Node at df302265-7ca7-da7c-d545-835cf35c578b [Follower]" leader-address= leader-id= -2024-07-05T13:30:03.840Z [WARN] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:03.840Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: entering candidate state: node="Node at 930cee5f-afaa-ee8c-62e1-4c981d8a51bc [Candidate]" term=2 -2024-07-05T13:30:03.840Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: election won: term=2 tally=2 -2024-07-05T13:30:03.840Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: entering leader state: leader="Node at 930cee5f-afaa-ee8c-62e1-4c981d8a51bc [Leader]" -2024-07-05T13:30:03.840Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: added peer, starting replication: peer=server-0701f8bc-7092-53a8-ffb6-e79314e819c8 -2024-07-05T13:30:03.840Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: added peer, starting replication: peer=server-df302265-7ca7-da7c-d545-835cf35c578b -2024-07-05T13:30:03.840Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: pipelining replication: peer="{Voter server-df302265-7ca7-da7c-d545-835cf35c578b df302265-7ca7-da7c-d545-835cf35c578b}" -2024-07-05T13:30:03.841Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: pipelining replication: peer="{Voter server-0701f8bc-7092-53a8-ffb6-e79314e819c8 0701f8bc-7092-53a8-ffb6-e79314e819c8}" -2024-07-05T13:30:04.846Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:03.841416389 +0000 UTC m=+44.611327064 (1 nodes), 547.725905ms from start of poll, 548.409041ms from cluster start. Timeout at 2024-07-05 13:30:04.846773439 +0000 UTC m=+45.616684114, 1.00535705s after stability -2024-07-05T13:30:04.848Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: starting snapshot up to: index=12 -2024-07-05T13:30:04.848Z [INFO] creating new snapshot: path=/tmp/raft2728261157/snapshots/2-12-1720186204848.tmp -2024-07-05T13:30:04.850Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: snapshot complete up to: index=12 -2024-07-05T13:30:04.852Z [INFO] creating new snapshot: path=/tmp/raft2728261157/snapshots/2-23-1720186204852.tmp -2024-07-05T13:30:04.855Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: copied to local snapshot: bytes=71 -2024-07-05T13:30:04.861Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: snapshot restore progress: id=2-23-1720186204852 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:04.861Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: removing all old logs from log store -2024-07-05T13:30:04.861Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: compacting logs: from=1 to=22 -2024-07-05T13:30:04.861Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: restored user snapshot: index=23 -2024-07-05T13:30:04.862Z [ERROR] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: failed to get log: index=22 error="log not found" -2024-07-05T13:30:04.862Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: aborting pipeline replication: peer="{Voter server-df302265-7ca7-da7c-d545-835cf35c578b df302265-7ca7-da7c-d545-835cf35c578b}" -2024-07-05T13:30:04.862Z [ERROR] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: failed to get log: index=22 error="log not found" -2024-07-05T13:30:04.862Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: aborting pipeline replication: peer="{Voter server-0701f8bc-7092-53a8-ffb6-e79314e819c8 0701f8bc-7092-53a8-ffb6-e79314e819c8}" -2024-07-05T13:30:04.869Z [ERROR] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: failed to get log: index=22 error="log not found" -2024-07-05T13:30:04.870Z [INFO] creating new snapshot: path=/tmp/raft3998919/snapshots/2-23-1720186204870.tmp -2024-07-05T13:30:04.870Z [INFO] server-0701f8bc-7092-53a8-ffb6-e79314e819c8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:04.871Z [INFO] server-0701f8bc-7092-53a8-ffb6-e79314e819c8: copied to local snapshot: bytes=71 -2024-07-05T13:30:04.871Z [ERROR] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: failed to get log: index=22 error="log not found" -2024-07-05T13:30:04.872Z [INFO] creating new snapshot: path=/tmp/raft4086892609/snapshots/2-23-1720186204872.tmp -2024-07-05T13:30:04.872Z [INFO] server-df302265-7ca7-da7c-d545-835cf35c578b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:04.873Z [INFO] server-df302265-7ca7-da7c-d545-835cf35c578b: copied to local snapshot: bytes=71 -2024-07-05T13:30:04.874Z [INFO] server-0701f8bc-7092-53a8-ffb6-e79314e819c8: snapshot restore progress: id=2-23-1720186204870 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:04.874Z [INFO] server-0701f8bc-7092-53a8-ffb6-e79314e819c8: removing all old logs from log store -2024-07-05T13:30:04.874Z [INFO] server-0701f8bc-7092-53a8-ffb6-e79314e819c8: compacting logs: from=1 to=22 -2024-07-05T13:30:04.874Z [INFO] server-0701f8bc-7092-53a8-ffb6-e79314e819c8: Installed remote snapshot -2024-07-05T13:30:04.874Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: pipelining replication: peer="{Voter server-0701f8bc-7092-53a8-ffb6-e79314e819c8 0701f8bc-7092-53a8-ffb6-e79314e819c8}" -2024-07-05T13:30:04.874Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:04.874Z [INFO] server-df302265-7ca7-da7c-d545-835cf35c578b: snapshot restore progress: id=2-23-1720186204872 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:04.875Z [INFO] server-df302265-7ca7-da7c-d545-835cf35c578b: removing all old logs from log store -2024-07-05T13:30:04.875Z [INFO] server-df302265-7ca7-da7c-d545-835cf35c578b: compacting logs: from=1 to=22 -2024-07-05T13:30:04.875Z [INFO] server-df302265-7ca7-da7c-d545-835cf35c578b: Installed remote snapshot -2024-07-05T13:30:04.875Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: pipelining replication: peer="{Voter server-df302265-7ca7-da7c-d545-835cf35c578b df302265-7ca7-da7c-d545-835cf35c578b}" -2024-07-05T13:30:05.880Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:04.874550184 +0000 UTC m=+45.644460859 (2 nodes), 1.6µs from start of poll, 1.581542836s from cluster start. Timeout at 2024-07-05 13:30:05.880286261 +0000 UTC m=+46.650196936, 1.005736077s after stability -2024-07-05T13:30:05.893Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: aborting pipeline replication: peer="{Voter server-df302265-7ca7-da7c-d545-835cf35c578b df302265-7ca7-da7c-d545-835cf35c578b}" -2024-07-05T13:30:05.893Z [INFO] server-930cee5f-afaa-ee8c-62e1-4c981d8a51bc: aborting pipeline replication: peer="{Voter server-0701f8bc-7092-53a8-ffb6-e79314e819c8 0701f8bc-7092-53a8-ffb6-e79314e819c8}" +2024-07-05T13:36:20.072Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b5de95e-7ce3-438e-1f58-20db3609f69d Address:9b5de95e-7ce3-438e-1f58-20db3609f69d} {Suffrage:Voter ID:server-8604a2ee-e3fe-a313-5674-077adbb8fac8 Address:8604a2ee-e3fe-a313-5674-077adbb8fac8} {Suffrage:Voter ID:server-99a96b39-de62-5267-d226-be74f01425c9 Address:99a96b39-de62-5267-d226-be74f01425c9}]" +2024-07-05T13:36:20.073Z [INFO] server-8604a2ee-e3fe-a313-5674-077adbb8fac8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b5de95e-7ce3-438e-1f58-20db3609f69d Address:9b5de95e-7ce3-438e-1f58-20db3609f69d} {Suffrage:Voter ID:server-8604a2ee-e3fe-a313-5674-077adbb8fac8 Address:8604a2ee-e3fe-a313-5674-077adbb8fac8} {Suffrage:Voter ID:server-99a96b39-de62-5267-d226-be74f01425c9 Address:99a96b39-de62-5267-d226-be74f01425c9}]" +2024-07-05T13:36:20.073Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: entering follower state: follower="Node at 9b5de95e-7ce3-438e-1f58-20db3609f69d [Follower]" leader-address= leader-id= +2024-07-05T13:36:20.074Z [INFO] server-99a96b39-de62-5267-d226-be74f01425c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b5de95e-7ce3-438e-1f58-20db3609f69d Address:9b5de95e-7ce3-438e-1f58-20db3609f69d} {Suffrage:Voter ID:server-8604a2ee-e3fe-a313-5674-077adbb8fac8 Address:8604a2ee-e3fe-a313-5674-077adbb8fac8} {Suffrage:Voter ID:server-99a96b39-de62-5267-d226-be74f01425c9 Address:99a96b39-de62-5267-d226-be74f01425c9}]" +2024-07-05T13:36:20.074Z [INFO] server-8604a2ee-e3fe-a313-5674-077adbb8fac8: entering follower state: follower="Node at 8604a2ee-e3fe-a313-5674-077adbb8fac8 [Follower]" leader-address= leader-id= +2024-07-05T13:36:20.074Z [INFO] server-99a96b39-de62-5267-d226-be74f01425c9: entering follower state: follower="Node at 99a96b39-de62-5267-d226-be74f01425c9 [Follower]" leader-address= leader-id= +2024-07-05T13:36:20.074Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:20.678Z [WARN] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:20.679Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: entering candidate state: node="Node at 9b5de95e-7ce3-438e-1f58-20db3609f69d [Candidate]" term=2 +2024-07-05T13:36:20.679Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: election won: term=2 tally=2 +2024-07-05T13:36:20.679Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: entering leader state: leader="Node at 9b5de95e-7ce3-438e-1f58-20db3609f69d [Leader]" +2024-07-05T13:36:20.680Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: added peer, starting replication: peer=server-8604a2ee-e3fe-a313-5674-077adbb8fac8 +2024-07-05T13:36:20.680Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: added peer, starting replication: peer=server-99a96b39-de62-5267-d226-be74f01425c9 +2024-07-05T13:36:20.680Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: pipelining replication: peer="{Voter server-8604a2ee-e3fe-a313-5674-077adbb8fac8 8604a2ee-e3fe-a313-5674-077adbb8fac8}" +2024-07-05T13:36:20.681Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: pipelining replication: peer="{Voter server-99a96b39-de62-5267-d226-be74f01425c9 99a96b39-de62-5267-d226-be74f01425c9}" +2024-07-05T13:36:21.685Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:20.680121091 +0000 UTC m=+46.676042543 (1 nodes), 605.756978ms from start of poll, 608.293849ms from cluster start. Timeout at 2024-07-05 13:36:21.68529019 +0000 UTC m=+47.681211642, 1.005169099s after stability +2024-07-05T13:36:21.690Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: starting snapshot up to: index=12 +2024-07-05T13:36:21.691Z [INFO] creating new snapshot: path=/tmp/raft3365990027/snapshots/2-12-1720186581691.tmp +2024-07-05T13:36:21.695Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: snapshot complete up to: index=12 +2024-07-05T13:36:21.696Z [INFO] creating new snapshot: path=/tmp/raft3365990027/snapshots/2-23-1720186581696.tmp +2024-07-05T13:36:21.698Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: copied to local snapshot: bytes=71 +2024-07-05T13:36:21.698Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: snapshot restore progress: id=2-23-1720186581696 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:21.698Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: removing all old logs from log store +2024-07-05T13:36:21.698Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: compacting logs: from=1 to=22 +2024-07-05T13:36:21.698Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: restored user snapshot: index=23 +2024-07-05T13:36:21.698Z [ERROR] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: failed to get log: index=22 error="log not found" +2024-07-05T13:36:21.698Z [ERROR] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: failed to get log: index=22 error="log not found" +2024-07-05T13:36:21.698Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: aborting pipeline replication: peer="{Voter server-8604a2ee-e3fe-a313-5674-077adbb8fac8 8604a2ee-e3fe-a313-5674-077adbb8fac8}" +2024-07-05T13:36:21.698Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: aborting pipeline replication: peer="{Voter server-99a96b39-de62-5267-d226-be74f01425c9 99a96b39-de62-5267-d226-be74f01425c9}" +2024-07-05T13:36:21.705Z [ERROR] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: failed to get log: index=22 error="log not found" +2024-07-05T13:36:21.708Z [INFO] creating new snapshot: path=/tmp/raft2445287221/snapshots/2-23-1720186581708.tmp +2024-07-05T13:36:21.710Z [ERROR] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: failed to get log: index=22 error="log not found" +2024-07-05T13:36:21.711Z [INFO] server-99a96b39-de62-5267-d226-be74f01425c9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:21.712Z [INFO] creating new snapshot: path=/tmp/raft3474833925/snapshots/2-23-1720186581712.tmp +2024-07-05T13:36:21.713Z [INFO] server-99a96b39-de62-5267-d226-be74f01425c9: copied to local snapshot: bytes=71 +2024-07-05T13:36:21.713Z [INFO] server-8604a2ee-e3fe-a313-5674-077adbb8fac8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:21.714Z [INFO] server-99a96b39-de62-5267-d226-be74f01425c9: snapshot restore progress: id=2-23-1720186581708 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:21.714Z [INFO] server-99a96b39-de62-5267-d226-be74f01425c9: removing all old logs from log store +2024-07-05T13:36:21.714Z [INFO] server-99a96b39-de62-5267-d226-be74f01425c9: compacting logs: from=1 to=22 +2024-07-05T13:36:21.714Z [INFO] server-99a96b39-de62-5267-d226-be74f01425c9: Installed remote snapshot +2024-07-05T13:36:21.715Z [INFO] server-8604a2ee-e3fe-a313-5674-077adbb8fac8: copied to local snapshot: bytes=71 +2024-07-05T13:36:21.715Z [WARN] server-99a96b39-de62-5267-d226-be74f01425c9: skipping application of old log: index=22 +2024-07-05T13:36:21.715Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: pipelining replication: peer="{Voter server-99a96b39-de62-5267-d226-be74f01425c9 99a96b39-de62-5267-d226-be74f01425c9}" +2024-07-05T13:36:21.715Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:36:21.716Z [INFO] server-8604a2ee-e3fe-a313-5674-077adbb8fac8: snapshot restore progress: id=2-23-1720186581712 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:21.716Z [INFO] server-8604a2ee-e3fe-a313-5674-077adbb8fac8: removing all old logs from log store +2024-07-05T13:36:21.716Z [INFO] server-8604a2ee-e3fe-a313-5674-077adbb8fac8: compacting logs: from=1 to=22 +2024-07-05T13:36:21.716Z [INFO] server-8604a2ee-e3fe-a313-5674-077adbb8fac8: Installed remote snapshot +2024-07-05T13:36:21.717Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: pipelining replication: peer="{Voter server-8604a2ee-e3fe-a313-5674-077adbb8fac8 8604a2ee-e3fe-a313-5674-077adbb8fac8}" +2024-07-05T13:36:22.721Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:36:21.715908214 +0000 UTC m=+47.711829083 (2 nodes), 5.542µs from start of poll, 1.644080389s from cluster start. Timeout at 2024-07-05 13:36:22.721130688 +0000 UTC m=+48.717052140, 1.005223057s after stability +2024-07-05T13:36:22.737Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: aborting pipeline replication: peer="{Voter server-8604a2ee-e3fe-a313-5674-077adbb8fac8 8604a2ee-e3fe-a313-5674-077adbb8fac8}" +2024-07-05T13:36:22.737Z [INFO] server-9b5de95e-7ce3-438e-1f58-20db3609f69d: aborting pipeline replication: peer="{Voter server-99a96b39-de62-5267-d226-be74f01425c9 99a96b39-de62-5267-d226-be74f01425c9}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true -2024-07-05T13:30:05.898Z [INFO] server-204d1e8e-3c63-f798-4d6f-d4700693c122: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-204d1e8e-3c63-f798-4d6f-d4700693c122 Address:204d1e8e-3c63-f798-4d6f-d4700693c122} {Suffrage:Voter ID:server-7f913cbd-889c-3fd5-e203-4dca421675a7 Address:7f913cbd-889c-3fd5-e203-4dca421675a7} {Suffrage:Voter ID:server-e1c008d4-2ccf-aa71-00e7-388488a12dc7 Address:e1c008d4-2ccf-aa71-00e7-388488a12dc7}]" -2024-07-05T13:30:05.899Z [INFO] server-204d1e8e-3c63-f798-4d6f-d4700693c122: entering follower state: follower="Node at 204d1e8e-3c63-f798-4d6f-d4700693c122 [Follower]" leader-address= leader-id= -2024-07-05T13:30:05.899Z [INFO] server-7f913cbd-889c-3fd5-e203-4dca421675a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-204d1e8e-3c63-f798-4d6f-d4700693c122 Address:204d1e8e-3c63-f798-4d6f-d4700693c122} {Suffrage:Voter ID:server-7f913cbd-889c-3fd5-e203-4dca421675a7 Address:7f913cbd-889c-3fd5-e203-4dca421675a7} {Suffrage:Voter ID:server-e1c008d4-2ccf-aa71-00e7-388488a12dc7 Address:e1c008d4-2ccf-aa71-00e7-388488a12dc7}]" -2024-07-05T13:30:05.899Z [INFO] server-7f913cbd-889c-3fd5-e203-4dca421675a7: entering follower state: follower="Node at 7f913cbd-889c-3fd5-e203-4dca421675a7 [Follower]" leader-address= leader-id= -2024-07-05T13:30:05.899Z [INFO] server-e1c008d4-2ccf-aa71-00e7-388488a12dc7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-204d1e8e-3c63-f798-4d6f-d4700693c122 Address:204d1e8e-3c63-f798-4d6f-d4700693c122} {Suffrage:Voter ID:server-7f913cbd-889c-3fd5-e203-4dca421675a7 Address:7f913cbd-889c-3fd5-e203-4dca421675a7} {Suffrage:Voter ID:server-e1c008d4-2ccf-aa71-00e7-388488a12dc7 Address:e1c008d4-2ccf-aa71-00e7-388488a12dc7}]" -2024-07-05T13:30:05.899Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:05.899Z [INFO] server-e1c008d4-2ccf-aa71-00e7-388488a12dc7: entering follower state: follower="Node at e1c008d4-2ccf-aa71-00e7-388488a12dc7 [Follower]" leader-address= leader-id= -2024-07-05T13:30:06.516Z [WARN] server-7f913cbd-889c-3fd5-e203-4dca421675a7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:06.516Z [INFO] server-7f913cbd-889c-3fd5-e203-4dca421675a7: entering candidate state: node="Node at 7f913cbd-889c-3fd5-e203-4dca421675a7 [Candidate]" term=2 -2024-07-05T13:30:06.516Z [INFO] server-7f913cbd-889c-3fd5-e203-4dca421675a7: election won: term=2 tally=2 -2024-07-05T13:30:06.516Z [INFO] server-7f913cbd-889c-3fd5-e203-4dca421675a7: entering leader state: leader="Node at 7f913cbd-889c-3fd5-e203-4dca421675a7 [Leader]" -2024-07-05T13:30:06.516Z [INFO] server-7f913cbd-889c-3fd5-e203-4dca421675a7: added peer, starting replication: peer=server-204d1e8e-3c63-f798-4d6f-d4700693c122 -2024-07-05T13:30:06.517Z [INFO] server-7f913cbd-889c-3fd5-e203-4dca421675a7: added peer, starting replication: peer=server-e1c008d4-2ccf-aa71-00e7-388488a12dc7 -2024-07-05T13:30:06.517Z [INFO] server-7f913cbd-889c-3fd5-e203-4dca421675a7: pipelining replication: peer="{Voter server-e1c008d4-2ccf-aa71-00e7-388488a12dc7 e1c008d4-2ccf-aa71-00e7-388488a12dc7}" -2024-07-05T13:30:06.517Z [INFO] server-7f913cbd-889c-3fd5-e203-4dca421675a7: pipelining replication: peer="{Voter server-204d1e8e-3c63-f798-4d6f-d4700693c122 204d1e8e-3c63-f798-4d6f-d4700693c122}" -2024-07-05T13:30:07.524Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:06.517572983 +0000 UTC m=+47.287483658 (1 nodes), 618.080036ms from start of poll, 619.185757ms from cluster start. Timeout at 2024-07-05 13:30:07.524220108 +0000 UTC m=+48.294130823, 1.006647165s after stability -2024-07-05T13:30:07.528Z [INFO] server-7f913cbd-889c-3fd5-e203-4dca421675a7: starting snapshot up to: index=12 -2024-07-05T13:30:07.528Z [INFO] creating new snapshot: path=/tmp/raft2341888356/snapshots/2-12-1720186207528.tmp -2024-07-05T13:30:07.530Z [INFO] server-7f913cbd-889c-3fd5-e203-4dca421675a7: snapshot complete up to: index=12 -2024-07-05T13:30:07.536Z [INFO] server-63103ded-b50a-bc88-8adc-8e111be0ef18: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63103ded-b50a-bc88-8adc-8e111be0ef18 Address:63103ded-b50a-bc88-8adc-8e111be0ef18} {Suffrage:Voter ID:server-257222e1-c135-9c25-d0b0-f928b7c5da29 Address:257222e1-c135-9c25-d0b0-f928b7c5da29} {Suffrage:Voter ID:server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2 Address:e8726ce6-7eab-6f05-fe6b-0c087068dcf2}]" -2024-07-05T13:30:07.537Z [INFO] server-63103ded-b50a-bc88-8adc-8e111be0ef18: entering follower state: follower="Node at 63103ded-b50a-bc88-8adc-8e111be0ef18 [Follower]" leader-address= leader-id= -2024-07-05T13:30:07.537Z [INFO] server-257222e1-c135-9c25-d0b0-f928b7c5da29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63103ded-b50a-bc88-8adc-8e111be0ef18 Address:63103ded-b50a-bc88-8adc-8e111be0ef18} {Suffrage:Voter ID:server-257222e1-c135-9c25-d0b0-f928b7c5da29 Address:257222e1-c135-9c25-d0b0-f928b7c5da29} {Suffrage:Voter ID:server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2 Address:e8726ce6-7eab-6f05-fe6b-0c087068dcf2}]" -2024-07-05T13:30:07.537Z [INFO] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63103ded-b50a-bc88-8adc-8e111be0ef18 Address:63103ded-b50a-bc88-8adc-8e111be0ef18} {Suffrage:Voter ID:server-257222e1-c135-9c25-d0b0-f928b7c5da29 Address:257222e1-c135-9c25-d0b0-f928b7c5da29} {Suffrage:Voter ID:server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2 Address:e8726ce6-7eab-6f05-fe6b-0c087068dcf2}]" -2024-07-05T13:30:07.537Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:07.537Z [INFO] server-257222e1-c135-9c25-d0b0-f928b7c5da29: entering follower state: follower="Node at 257222e1-c135-9c25-d0b0-f928b7c5da29 [Follower]" leader-address= leader-id= -2024-07-05T13:30:07.537Z [INFO] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: entering follower state: follower="Node at e8726ce6-7eab-6f05-fe6b-0c087068dcf2 [Follower]" leader-address= leader-id= -2024-07-05T13:30:08.215Z [WARN] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:08.215Z [INFO] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: entering candidate state: node="Node at e8726ce6-7eab-6f05-fe6b-0c087068dcf2 [Candidate]" term=2 -2024-07-05T13:30:08.215Z [INFO] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: election won: term=2 tally=2 -2024-07-05T13:30:08.215Z [INFO] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: entering leader state: leader="Node at e8726ce6-7eab-6f05-fe6b-0c087068dcf2 [Leader]" -2024-07-05T13:30:08.216Z [INFO] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: added peer, starting replication: peer=server-63103ded-b50a-bc88-8adc-8e111be0ef18 -2024-07-05T13:30:08.216Z [INFO] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: added peer, starting replication: peer=server-257222e1-c135-9c25-d0b0-f928b7c5da29 -2024-07-05T13:30:08.216Z [INFO] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: pipelining replication: peer="{Voter server-257222e1-c135-9c25-d0b0-f928b7c5da29 257222e1-c135-9c25-d0b0-f928b7c5da29}" -2024-07-05T13:30:08.216Z [INFO] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: pipelining replication: peer="{Voter server-63103ded-b50a-bc88-8adc-8e111be0ef18 63103ded-b50a-bc88-8adc-8e111be0ef18}" -2024-07-05T13:30:09.222Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:08.216590738 +0000 UTC m=+48.986501413 (1 nodes), 679.119815ms from start of poll, 681.440694ms from cluster start. Timeout at 2024-07-05 13:30:09.222071863 +0000 UTC m=+49.991982538, 1.005481125s after stability -2024-07-05T13:30:09.222Z [INFO] creating new snapshot: path=/tmp/raft4168669023/snapshots/2-15-1720186209222.tmp -2024-07-05T13:30:09.223Z [INFO] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: copied to local snapshot: bytes=71 -2024-07-05T13:30:09.224Z [INFO] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: snapshot restore progress: id=2-15-1720186209222 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:09.224Z [INFO] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: restored user snapshot: index=15 -2024-07-05T13:30:09.224Z [ERROR] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: failed to get log: index=3 error="log not found" -2024-07-05T13:30:09.224Z [INFO] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: aborting pipeline replication: peer="{Voter server-257222e1-c135-9c25-d0b0-f928b7c5da29 257222e1-c135-9c25-d0b0-f928b7c5da29}" -2024-07-05T13:30:09.224Z [ERROR] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: failed to get log: index=3 error="log not found" -2024-07-05T13:30:09.224Z [INFO] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: aborting pipeline replication: peer="{Voter server-63103ded-b50a-bc88-8adc-8e111be0ef18 63103ded-b50a-bc88-8adc-8e111be0ef18}" -2024-07-05T13:30:09.232Z [ERROR] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: failed to get log: index=3 error="log not found" -2024-07-05T13:30:09.232Z [INFO] creating new snapshot: path=/tmp/raft2418087217/snapshots/2-15-1720186209232.tmp -2024-07-05T13:30:09.233Z [INFO] server-257222e1-c135-9c25-d0b0-f928b7c5da29: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:09.234Z [INFO] server-257222e1-c135-9c25-d0b0-f928b7c5da29: copied to local snapshot: bytes=71 -2024-07-05T13:30:09.234Z [INFO] server-257222e1-c135-9c25-d0b0-f928b7c5da29: snapshot restore progress: id=2-15-1720186209232 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:09.234Z [INFO] server-257222e1-c135-9c25-d0b0-f928b7c5da29: Installed remote snapshot -2024-07-05T13:30:09.234Z [INFO] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: pipelining replication: peer="{Voter server-257222e1-c135-9c25-d0b0-f928b7c5da29 257222e1-c135-9c25-d0b0-f928b7c5da29}" -2024-07-05T13:30:09.234Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:09.234Z [ERROR] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: failed to get log: index=3 error="log not found" -2024-07-05T13:30:09.235Z [INFO] creating new snapshot: path=/tmp/raft3243661119/snapshots/2-15-1720186209235.tmp -2024-07-05T13:30:09.235Z [INFO] server-63103ded-b50a-bc88-8adc-8e111be0ef18: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:09.236Z [INFO] server-63103ded-b50a-bc88-8adc-8e111be0ef18: copied to local snapshot: bytes=71 -2024-07-05T13:30:09.237Z [INFO] server-63103ded-b50a-bc88-8adc-8e111be0ef18: snapshot restore progress: id=2-15-1720186209235 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:09.237Z [INFO] server-63103ded-b50a-bc88-8adc-8e111be0ef18: Installed remote snapshot -2024-07-05T13:30:09.237Z [INFO] server-e8726ce6-7eab-6f05-fe6b-0c087068dcf2: pipelining replication: peer="{Voter server-63103ded-b50a-bc88-8adc-8e111be0ef18 63103ded-b50a-bc88-8adc-8e111be0ef18}" -2024-07-05T13:30:10.240Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:09.234913213 +0000 UTC m=+50.004823888 (2 nodes), 1.56µs from start of poll, 1.699763169s from cluster start. Timeout at 2024-07-05 13:30:10.240259423 +0000 UTC m=+51.010170098, 1.00534621s after stability -2024-07-05T13:30:10.254Z [INFO] server-7f913cbd-889c-3fd5-e203-4dca421675a7: aborting pipeline replication: peer="{Voter server-e1c008d4-2ccf-aa71-00e7-388488a12dc7 e1c008d4-2ccf-aa71-00e7-388488a12dc7}" -2024-07-05T13:30:10.254Z [INFO] server-7f913cbd-889c-3fd5-e203-4dca421675a7: aborting pipeline replication: peer="{Voter server-204d1e8e-3c63-f798-4d6f-d4700693c122 204d1e8e-3c63-f798-4d6f-d4700693c122}" +2024-07-05T13:36:22.752Z [INFO] server-0d2a64a0-68fd-5f6c-691c-3187f63dae9c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d2a64a0-68fd-5f6c-691c-3187f63dae9c Address:0d2a64a0-68fd-5f6c-691c-3187f63dae9c} {Suffrage:Voter ID:server-392f52ab-9297-4226-3940-534a2e1393aa Address:392f52ab-9297-4226-3940-534a2e1393aa} {Suffrage:Voter ID:server-bdb4602f-c75e-7328-aca3-a367a32410ec Address:bdb4602f-c75e-7328-aca3-a367a32410ec}]" +2024-07-05T13:36:22.753Z [INFO] server-0d2a64a0-68fd-5f6c-691c-3187f63dae9c: entering follower state: follower="Node at 0d2a64a0-68fd-5f6c-691c-3187f63dae9c [Follower]" leader-address= leader-id= +2024-07-05T13:36:22.753Z [INFO] server-392f52ab-9297-4226-3940-534a2e1393aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d2a64a0-68fd-5f6c-691c-3187f63dae9c Address:0d2a64a0-68fd-5f6c-691c-3187f63dae9c} {Suffrage:Voter ID:server-392f52ab-9297-4226-3940-534a2e1393aa Address:392f52ab-9297-4226-3940-534a2e1393aa} {Suffrage:Voter ID:server-bdb4602f-c75e-7328-aca3-a367a32410ec Address:bdb4602f-c75e-7328-aca3-a367a32410ec}]" +2024-07-05T13:36:22.754Z [INFO] server-392f52ab-9297-4226-3940-534a2e1393aa: entering follower state: follower="Node at 392f52ab-9297-4226-3940-534a2e1393aa [Follower]" leader-address= leader-id= +2024-07-05T13:36:22.754Z [INFO] server-bdb4602f-c75e-7328-aca3-a367a32410ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d2a64a0-68fd-5f6c-691c-3187f63dae9c Address:0d2a64a0-68fd-5f6c-691c-3187f63dae9c} {Suffrage:Voter ID:server-392f52ab-9297-4226-3940-534a2e1393aa Address:392f52ab-9297-4226-3940-534a2e1393aa} {Suffrage:Voter ID:server-bdb4602f-c75e-7328-aca3-a367a32410ec Address:bdb4602f-c75e-7328-aca3-a367a32410ec}]" +2024-07-05T13:36:22.755Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:22.755Z [INFO] server-bdb4602f-c75e-7328-aca3-a367a32410ec: entering follower state: follower="Node at bdb4602f-c75e-7328-aca3-a367a32410ec [Follower]" leader-address= leader-id= +2024-07-05T13:36:23.360Z [WARN] server-392f52ab-9297-4226-3940-534a2e1393aa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:23.360Z [INFO] server-392f52ab-9297-4226-3940-534a2e1393aa: entering candidate state: node="Node at 392f52ab-9297-4226-3940-534a2e1393aa [Candidate]" term=2 +2024-07-05T13:36:23.361Z [INFO] server-392f52ab-9297-4226-3940-534a2e1393aa: election won: term=2 tally=2 +2024-07-05T13:36:23.361Z [INFO] server-392f52ab-9297-4226-3940-534a2e1393aa: entering leader state: leader="Node at 392f52ab-9297-4226-3940-534a2e1393aa [Leader]" +2024-07-05T13:36:23.361Z [INFO] server-392f52ab-9297-4226-3940-534a2e1393aa: added peer, starting replication: peer=server-0d2a64a0-68fd-5f6c-691c-3187f63dae9c +2024-07-05T13:36:23.362Z [INFO] server-392f52ab-9297-4226-3940-534a2e1393aa: added peer, starting replication: peer=server-bdb4602f-c75e-7328-aca3-a367a32410ec +2024-07-05T13:36:23.362Z [INFO] server-392f52ab-9297-4226-3940-534a2e1393aa: pipelining replication: peer="{Voter server-0d2a64a0-68fd-5f6c-691c-3187f63dae9c 0d2a64a0-68fd-5f6c-691c-3187f63dae9c}" +2024-07-05T13:36:23.362Z [INFO] server-392f52ab-9297-4226-3940-534a2e1393aa: pipelining replication: peer="{Voter server-bdb4602f-c75e-7328-aca3-a367a32410ec bdb4602f-c75e-7328-aca3-a367a32410ec}" +2024-07-05T13:36:24.367Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:23.362236145 +0000 UTC m=+49.358157597 (1 nodes), 607.027746ms from start of poll, 609.958068ms from cluster start. Timeout at 2024-07-05 13:36:24.367598033 +0000 UTC m=+50.363519193, 1.005361596s after stability +2024-07-05T13:36:24.372Z [INFO] server-392f52ab-9297-4226-3940-534a2e1393aa: starting snapshot up to: index=12 +2024-07-05T13:36:24.372Z [INFO] creating new snapshot: path=/tmp/raft1461167715/snapshots/2-12-1720186584372.tmp +2024-07-05T13:36:24.380Z [INFO] server-392f52ab-9297-4226-3940-534a2e1393aa: snapshot complete up to: index=12 +2024-07-05T13:36:24.393Z [INFO] server-11d8e47e-e734-eb0e-beef-573a5ac6bf3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d8e47e-e734-eb0e-beef-573a5ac6bf3f Address:11d8e47e-e734-eb0e-beef-573a5ac6bf3f} {Suffrage:Voter ID:server-7d45c949-f6ea-de17-8812-4c8100aa89ea Address:7d45c949-f6ea-de17-8812-4c8100aa89ea} {Suffrage:Voter ID:server-ced3d4e4-1ae4-1218-cae3-eb41cc3ec399 Address:ced3d4e4-1ae4-1218-cae3-eb41cc3ec399}]" +2024-07-05T13:36:24.393Z [INFO] server-11d8e47e-e734-eb0e-beef-573a5ac6bf3f: entering follower state: follower="Node at 11d8e47e-e734-eb0e-beef-573a5ac6bf3f [Follower]" leader-address= leader-id= +2024-07-05T13:36:24.394Z [INFO] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d8e47e-e734-eb0e-beef-573a5ac6bf3f Address:11d8e47e-e734-eb0e-beef-573a5ac6bf3f} {Suffrage:Voter ID:server-7d45c949-f6ea-de17-8812-4c8100aa89ea Address:7d45c949-f6ea-de17-8812-4c8100aa89ea} {Suffrage:Voter ID:server-ced3d4e4-1ae4-1218-cae3-eb41cc3ec399 Address:ced3d4e4-1ae4-1218-cae3-eb41cc3ec399}]" +2024-07-05T13:36:24.394Z [INFO] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: entering follower state: follower="Node at 7d45c949-f6ea-de17-8812-4c8100aa89ea [Follower]" leader-address= leader-id= +2024-07-05T13:36:24.395Z [INFO] server-ced3d4e4-1ae4-1218-cae3-eb41cc3ec399: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d8e47e-e734-eb0e-beef-573a5ac6bf3f Address:11d8e47e-e734-eb0e-beef-573a5ac6bf3f} {Suffrage:Voter ID:server-7d45c949-f6ea-de17-8812-4c8100aa89ea Address:7d45c949-f6ea-de17-8812-4c8100aa89ea} {Suffrage:Voter ID:server-ced3d4e4-1ae4-1218-cae3-eb41cc3ec399 Address:ced3d4e4-1ae4-1218-cae3-eb41cc3ec399}]" +2024-07-05T13:36:24.395Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:24.395Z [INFO] server-ced3d4e4-1ae4-1218-cae3-eb41cc3ec399: entering follower state: follower="Node at ced3d4e4-1ae4-1218-cae3-eb41cc3ec399 [Follower]" leader-address= leader-id= +2024-07-05T13:36:24.910Z [WARN] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:24.910Z [INFO] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: entering candidate state: node="Node at 7d45c949-f6ea-de17-8812-4c8100aa89ea [Candidate]" term=2 +2024-07-05T13:36:24.911Z [INFO] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: election won: term=2 tally=2 +2024-07-05T13:36:24.911Z [INFO] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: entering leader state: leader="Node at 7d45c949-f6ea-de17-8812-4c8100aa89ea [Leader]" +2024-07-05T13:36:24.911Z [INFO] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: added peer, starting replication: peer=server-11d8e47e-e734-eb0e-beef-573a5ac6bf3f +2024-07-05T13:36:24.911Z [INFO] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: added peer, starting replication: peer=server-ced3d4e4-1ae4-1218-cae3-eb41cc3ec399 +2024-07-05T13:36:24.912Z [INFO] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: pipelining replication: peer="{Voter server-11d8e47e-e734-eb0e-beef-573a5ac6bf3f 11d8e47e-e734-eb0e-beef-573a5ac6bf3f}" +2024-07-05T13:36:24.912Z [INFO] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: pipelining replication: peer="{Voter server-ced3d4e4-1ae4-1218-cae3-eb41cc3ec399 ced3d4e4-1ae4-1218-cae3-eb41cc3ec399}" +2024-07-05T13:36:25.916Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:24.91135831 +0000 UTC m=+50.907279762 (1 nodes), 515.799871ms from start of poll, 519.089519ms from cluster start. Timeout at 2024-07-05 13:36:25.916535868 +0000 UTC m=+51.912457028, 1.005177266s after stability +2024-07-05T13:36:25.918Z [INFO] creating new snapshot: path=/tmp/raft1586103830/snapshots/2-15-1720186585918.tmp +2024-07-05T13:36:25.925Z [INFO] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: copied to local snapshot: bytes=71 +2024-07-05T13:36:25.935Z [INFO] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: snapshot restore progress: id=2-15-1720186585918 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:25.935Z [INFO] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: restored user snapshot: index=15 +2024-07-05T13:36:25.935Z [ERROR] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: failed to get log: index=3 error="log not found" +2024-07-05T13:36:25.935Z [INFO] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: aborting pipeline replication: peer="{Voter server-ced3d4e4-1ae4-1218-cae3-eb41cc3ec399 ced3d4e4-1ae4-1218-cae3-eb41cc3ec399}" +2024-07-05T13:36:25.936Z [ERROR] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: failed to get log: index=3 error="log not found" +2024-07-05T13:36:25.936Z [INFO] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: aborting pipeline replication: peer="{Voter server-11d8e47e-e734-eb0e-beef-573a5ac6bf3f 11d8e47e-e734-eb0e-beef-573a5ac6bf3f}" +2024-07-05T13:36:25.942Z [ERROR] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: failed to get log: index=3 error="log not found" +2024-07-05T13:36:25.944Z [INFO] creating new snapshot: path=/tmp/raft1433996488/snapshots/2-15-1720186585944.tmp +2024-07-05T13:36:25.945Z [INFO] server-11d8e47e-e734-eb0e-beef-573a5ac6bf3f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:25.946Z [ERROR] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: failed to get log: index=3 error="log not found" +2024-07-05T13:36:25.947Z [INFO] server-11d8e47e-e734-eb0e-beef-573a5ac6bf3f: copied to local snapshot: bytes=71 +2024-07-05T13:36:25.948Z [INFO] creating new snapshot: path=/tmp/raft2870278996/snapshots/2-15-1720186585948.tmp +2024-07-05T13:36:25.948Z [INFO] server-11d8e47e-e734-eb0e-beef-573a5ac6bf3f: snapshot restore progress: id=2-15-1720186585944 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:25.948Z [INFO] server-11d8e47e-e734-eb0e-beef-573a5ac6bf3f: Installed remote snapshot +2024-07-05T13:36:25.949Z [INFO] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: pipelining replication: peer="{Voter server-11d8e47e-e734-eb0e-beef-573a5ac6bf3f 11d8e47e-e734-eb0e-beef-573a5ac6bf3f}" +2024-07-05T13:36:25.949Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:36:25.949Z [INFO] server-ced3d4e4-1ae4-1218-cae3-eb41cc3ec399: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:25.951Z [INFO] server-ced3d4e4-1ae4-1218-cae3-eb41cc3ec399: copied to local snapshot: bytes=71 +2024-07-05T13:36:25.952Z [INFO] server-ced3d4e4-1ae4-1218-cae3-eb41cc3ec399: snapshot restore progress: id=2-15-1720186585948 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:25.952Z [INFO] server-ced3d4e4-1ae4-1218-cae3-eb41cc3ec399: Installed remote snapshot +2024-07-05T13:36:25.953Z [INFO] server-7d45c949-f6ea-de17-8812-4c8100aa89ea: pipelining replication: peer="{Voter server-ced3d4e4-1ae4-1218-cae3-eb41cc3ec399 ced3d4e4-1ae4-1218-cae3-eb41cc3ec399}" +2024-07-05T13:36:26.955Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:36:25.949477558 +0000 UTC m=+51.945398427 (2 nodes), 5.25µs from start of poll, 1.557208184s from cluster start. Timeout at 2024-07-05 13:36:26.955163482 +0000 UTC m=+52.951084934, 1.005686507s after stability +2024-07-05T13:36:26.971Z [INFO] server-392f52ab-9297-4226-3940-534a2e1393aa: aborting pipeline replication: peer="{Voter server-0d2a64a0-68fd-5f6c-691c-3187f63dae9c 0d2a64a0-68fd-5f6c-691c-3187f63dae9c}" +2024-07-05T13:36:26.972Z [INFO] server-392f52ab-9297-4226-3940-534a2e1393aa: aborting pipeline replication: peer="{Voter server-bdb4602f-c75e-7328-aca3-a367a32410ec bdb4602f-c75e-7328-aca3-a367a32410ec}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true -2024-07-05T13:30:10.257Z [INFO] server-b4f37508-5084-eec8-0729-bd686d1fb9ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4f37508-5084-eec8-0729-bd686d1fb9ac Address:b4f37508-5084-eec8-0729-bd686d1fb9ac} {Suffrage:Voter ID:server-90b6308c-eda0-46d4-2269-3ae71d2c526a Address:90b6308c-eda0-46d4-2269-3ae71d2c526a} {Suffrage:Voter ID:server-c438b8d8-9354-013c-70a6-5050a597ba4f Address:c438b8d8-9354-013c-70a6-5050a597ba4f}]" -2024-07-05T13:30:10.257Z [INFO] server-90b6308c-eda0-46d4-2269-3ae71d2c526a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4f37508-5084-eec8-0729-bd686d1fb9ac Address:b4f37508-5084-eec8-0729-bd686d1fb9ac} {Suffrage:Voter ID:server-90b6308c-eda0-46d4-2269-3ae71d2c526a Address:90b6308c-eda0-46d4-2269-3ae71d2c526a} {Suffrage:Voter ID:server-c438b8d8-9354-013c-70a6-5050a597ba4f Address:c438b8d8-9354-013c-70a6-5050a597ba4f}]" -2024-07-05T13:30:10.257Z [INFO] server-c438b8d8-9354-013c-70a6-5050a597ba4f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4f37508-5084-eec8-0729-bd686d1fb9ac Address:b4f37508-5084-eec8-0729-bd686d1fb9ac} {Suffrage:Voter ID:server-90b6308c-eda0-46d4-2269-3ae71d2c526a Address:90b6308c-eda0-46d4-2269-3ae71d2c526a} {Suffrage:Voter ID:server-c438b8d8-9354-013c-70a6-5050a597ba4f Address:c438b8d8-9354-013c-70a6-5050a597ba4f}]" -2024-07-05T13:30:10.257Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:10.257Z [INFO] server-b4f37508-5084-eec8-0729-bd686d1fb9ac: entering follower state: follower="Node at b4f37508-5084-eec8-0729-bd686d1fb9ac [Follower]" leader-address= leader-id= -2024-07-05T13:30:10.257Z [INFO] server-90b6308c-eda0-46d4-2269-3ae71d2c526a: entering follower state: follower="Node at 90b6308c-eda0-46d4-2269-3ae71d2c526a [Follower]" leader-address= leader-id= -2024-07-05T13:30:10.257Z [INFO] server-c438b8d8-9354-013c-70a6-5050a597ba4f: entering follower state: follower="Node at c438b8d8-9354-013c-70a6-5050a597ba4f [Follower]" leader-address= leader-id= -2024-07-05T13:30:10.802Z [WARN] server-90b6308c-eda0-46d4-2269-3ae71d2c526a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:10.802Z [INFO] server-90b6308c-eda0-46d4-2269-3ae71d2c526a: entering candidate state: node="Node at 90b6308c-eda0-46d4-2269-3ae71d2c526a [Candidate]" term=2 -2024-07-05T13:30:10.802Z [INFO] server-90b6308c-eda0-46d4-2269-3ae71d2c526a: election won: term=2 tally=2 -2024-07-05T13:30:10.802Z [INFO] server-90b6308c-eda0-46d4-2269-3ae71d2c526a: entering leader state: leader="Node at 90b6308c-eda0-46d4-2269-3ae71d2c526a [Leader]" -2024-07-05T13:30:10.802Z [INFO] server-90b6308c-eda0-46d4-2269-3ae71d2c526a: added peer, starting replication: peer=server-b4f37508-5084-eec8-0729-bd686d1fb9ac -2024-07-05T13:30:10.802Z [INFO] server-90b6308c-eda0-46d4-2269-3ae71d2c526a: added peer, starting replication: peer=server-c438b8d8-9354-013c-70a6-5050a597ba4f -2024-07-05T13:30:10.802Z [INFO] server-90b6308c-eda0-46d4-2269-3ae71d2c526a: pipelining replication: peer="{Voter server-c438b8d8-9354-013c-70a6-5050a597ba4f c438b8d8-9354-013c-70a6-5050a597ba4f}" -2024-07-05T13:30:10.803Z [INFO] server-90b6308c-eda0-46d4-2269-3ae71d2c526a: pipelining replication: peer="{Voter server-b4f37508-5084-eec8-0729-bd686d1fb9ac b4f37508-5084-eec8-0729-bd686d1fb9ac}" -2024-07-05T13:30:11.808Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:10.803354308 +0000 UTC m=+51.573264983 (1 nodes), 545.742774ms from start of poll, 546.392551ms from cluster start. Timeout at 2024-07-05 13:30:11.808480126 +0000 UTC m=+52.578390801, 1.005125818s after stability -2024-07-05T13:30:11.810Z [INFO] server-90b6308c-eda0-46d4-2269-3ae71d2c526a: starting snapshot up to: index=12 -2024-07-05T13:30:11.810Z [INFO] creating new snapshot: path=/tmp/raft1809640358/snapshots/2-12-1720186211810.tmp -2024-07-05T13:30:11.811Z [INFO] server-90b6308c-eda0-46d4-2269-3ae71d2c526a: snapshot complete up to: index=12 -2024-07-05T13:30:11.813Z [INFO] server-52c20a7d-6030-13dc-35c2-02fae5094eea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-52c20a7d-6030-13dc-35c2-02fae5094eea Address:52c20a7d-6030-13dc-35c2-02fae5094eea} {Suffrage:Voter ID:server-c22dbc06-c6b0-ace1-f279-4ba0951c2590 Address:c22dbc06-c6b0-ace1-f279-4ba0951c2590} {Suffrage:Voter ID:server-1c95d167-f988-d644-0df0-20f51d0ff74c Address:1c95d167-f988-d644-0df0-20f51d0ff74c}]" -2024-07-05T13:30:11.815Z [INFO] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-52c20a7d-6030-13dc-35c2-02fae5094eea Address:52c20a7d-6030-13dc-35c2-02fae5094eea} {Suffrage:Voter ID:server-c22dbc06-c6b0-ace1-f279-4ba0951c2590 Address:c22dbc06-c6b0-ace1-f279-4ba0951c2590} {Suffrage:Voter ID:server-1c95d167-f988-d644-0df0-20f51d0ff74c Address:1c95d167-f988-d644-0df0-20f51d0ff74c}]" -2024-07-05T13:30:11.815Z [INFO] server-1c95d167-f988-d644-0df0-20f51d0ff74c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-52c20a7d-6030-13dc-35c2-02fae5094eea Address:52c20a7d-6030-13dc-35c2-02fae5094eea} {Suffrage:Voter ID:server-c22dbc06-c6b0-ace1-f279-4ba0951c2590 Address:c22dbc06-c6b0-ace1-f279-4ba0951c2590} {Suffrage:Voter ID:server-1c95d167-f988-d644-0df0-20f51d0ff74c Address:1c95d167-f988-d644-0df0-20f51d0ff74c}]" -2024-07-05T13:30:11.816Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:11.816Z [INFO] server-52c20a7d-6030-13dc-35c2-02fae5094eea: entering follower state: follower="Node at 52c20a7d-6030-13dc-35c2-02fae5094eea [Follower]" leader-address= leader-id= -2024-07-05T13:30:11.816Z [INFO] server-1c95d167-f988-d644-0df0-20f51d0ff74c: entering follower state: follower="Node at 1c95d167-f988-d644-0df0-20f51d0ff74c [Follower]" leader-address= leader-id= -2024-07-05T13:30:11.816Z [INFO] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: entering follower state: follower="Node at c22dbc06-c6b0-ace1-f279-4ba0951c2590 [Follower]" leader-address= leader-id= -2024-07-05T13:30:12.593Z [WARN] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:12.593Z [INFO] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: entering candidate state: node="Node at c22dbc06-c6b0-ace1-f279-4ba0951c2590 [Candidate]" term=2 -2024-07-05T13:30:12.593Z [INFO] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: election won: term=2 tally=2 -2024-07-05T13:30:12.593Z [INFO] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: entering leader state: leader="Node at c22dbc06-c6b0-ace1-f279-4ba0951c2590 [Leader]" -2024-07-05T13:30:12.593Z [INFO] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: added peer, starting replication: peer=server-52c20a7d-6030-13dc-35c2-02fae5094eea -2024-07-05T13:30:12.593Z [INFO] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: added peer, starting replication: peer=server-1c95d167-f988-d644-0df0-20f51d0ff74c -2024-07-05T13:30:12.594Z [INFO] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: pipelining replication: peer="{Voter server-52c20a7d-6030-13dc-35c2-02fae5094eea 52c20a7d-6030-13dc-35c2-02fae5094eea}" -2024-07-05T13:30:12.594Z [INFO] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: pipelining replication: peer="{Voter server-1c95d167-f988-d644-0df0-20f51d0ff74c 1c95d167-f988-d644-0df0-20f51d0ff74c}" -2024-07-05T13:30:13.600Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:12.594402634 +0000 UTC m=+53.364313309 (1 nodes), 778.334374ms from start of poll, 780.766809ms from cluster start. Timeout at 2024-07-05 13:30:13.600209348 +0000 UTC m=+54.370120023, 1.005806714s after stability -2024-07-05T13:30:13.600Z [INFO] creating new snapshot: path=/tmp/raft2623860800/snapshots/2-15-1720186213600.tmp -2024-07-05T13:30:13.601Z [INFO] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: copied to local snapshot: bytes=71 -2024-07-05T13:30:13.602Z [INFO] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: snapshot restore progress: id=2-15-1720186213600 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:13.602Z [INFO] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: removing all old logs from log store -2024-07-05T13:30:13.602Z [INFO] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: compacting logs: from=1 to=2 -2024-07-05T13:30:13.602Z [INFO] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: restored user snapshot: index=15 -2024-07-05T13:30:13.602Z [ERROR] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: failed to get log: index=2 error="log not found" -2024-07-05T13:30:13.602Z [INFO] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: aborting pipeline replication: peer="{Voter server-1c95d167-f988-d644-0df0-20f51d0ff74c 1c95d167-f988-d644-0df0-20f51d0ff74c}" -2024-07-05T13:30:13.602Z [ERROR] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: failed to get log: index=2 error="log not found" -2024-07-05T13:30:13.602Z [INFO] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: aborting pipeline replication: peer="{Voter server-52c20a7d-6030-13dc-35c2-02fae5094eea 52c20a7d-6030-13dc-35c2-02fae5094eea}" -2024-07-05T13:30:13.611Z [ERROR] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: failed to get log: index=2 error="log not found" -2024-07-05T13:30:13.612Z [INFO] creating new snapshot: path=/tmp/raft1112108970/snapshots/2-15-1720186213612.tmp -2024-07-05T13:30:13.612Z [INFO] server-1c95d167-f988-d644-0df0-20f51d0ff74c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:13.613Z [INFO] server-1c95d167-f988-d644-0df0-20f51d0ff74c: copied to local snapshot: bytes=71 -2024-07-05T13:30:13.613Z [INFO] server-1c95d167-f988-d644-0df0-20f51d0ff74c: snapshot restore progress: id=2-15-1720186213612 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:13.613Z [INFO] server-1c95d167-f988-d644-0df0-20f51d0ff74c: removing all old logs from log store -2024-07-05T13:30:13.613Z [INFO] server-1c95d167-f988-d644-0df0-20f51d0ff74c: compacting logs: from=1 to=2 -2024-07-05T13:30:13.613Z [INFO] server-1c95d167-f988-d644-0df0-20f51d0ff74c: Installed remote snapshot -2024-07-05T13:30:13.613Z [INFO] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: pipelining replication: peer="{Voter server-1c95d167-f988-d644-0df0-20f51d0ff74c 1c95d167-f988-d644-0df0-20f51d0ff74c}" -2024-07-05T13:30:13.614Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:13.614Z [ERROR] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: failed to get log: index=2 error="log not found" -2024-07-05T13:30:13.615Z [INFO] creating new snapshot: path=/tmp/raft726976213/snapshots/2-15-1720186213615.tmp -2024-07-05T13:30:13.615Z [INFO] server-52c20a7d-6030-13dc-35c2-02fae5094eea: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:13.616Z [INFO] server-52c20a7d-6030-13dc-35c2-02fae5094eea: copied to local snapshot: bytes=71 -2024-07-05T13:30:13.616Z [INFO] server-52c20a7d-6030-13dc-35c2-02fae5094eea: snapshot restore progress: id=2-15-1720186213615 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:13.616Z [INFO] server-52c20a7d-6030-13dc-35c2-02fae5094eea: removing all old logs from log store -2024-07-05T13:30:13.616Z [INFO] server-52c20a7d-6030-13dc-35c2-02fae5094eea: compacting logs: from=1 to=2 -2024-07-05T13:30:13.616Z [INFO] server-52c20a7d-6030-13dc-35c2-02fae5094eea: Installed remote snapshot -2024-07-05T13:30:13.616Z [INFO] server-c22dbc06-c6b0-ace1-f279-4ba0951c2590: pipelining replication: peer="{Voter server-52c20a7d-6030-13dc-35c2-02fae5094eea 52c20a7d-6030-13dc-35c2-02fae5094eea}" -2024-07-05T13:30:14.619Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:13.614245135 +0000 UTC m=+54.384155810 (2 nodes), 1.72µs from start of poll, 1.80060931s from cluster start. Timeout at 2024-07-05 13:30:14.619323195 +0000 UTC m=+55.389233870, 1.00507806s after stability -2024-07-05T13:30:14.633Z [INFO] server-90b6308c-eda0-46d4-2269-3ae71d2c526a: aborting pipeline replication: peer="{Voter server-b4f37508-5084-eec8-0729-bd686d1fb9ac b4f37508-5084-eec8-0729-bd686d1fb9ac}" -2024-07-05T13:30:14.633Z [INFO] server-90b6308c-eda0-46d4-2269-3ae71d2c526a: aborting pipeline replication: peer="{Voter server-c438b8d8-9354-013c-70a6-5050a597ba4f c438b8d8-9354-013c-70a6-5050a597ba4f}" +2024-07-05T13:36:26.984Z [INFO] server-8d25b4eb-1708-bf1f-fb80-60924d330379: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d25b4eb-1708-bf1f-fb80-60924d330379 Address:8d25b4eb-1708-bf1f-fb80-60924d330379} {Suffrage:Voter ID:server-0b4e2022-61b8-7450-4be2-915d2463f14e Address:0b4e2022-61b8-7450-4be2-915d2463f14e} {Suffrage:Voter ID:server-25c69b24-8f21-961b-07e1-1ca51ad034ed Address:25c69b24-8f21-961b-07e1-1ca51ad034ed}]" +2024-07-05T13:36:26.984Z [INFO] server-8d25b4eb-1708-bf1f-fb80-60924d330379: entering follower state: follower="Node at 8d25b4eb-1708-bf1f-fb80-60924d330379 [Follower]" leader-address= leader-id= +2024-07-05T13:36:26.985Z [INFO] server-0b4e2022-61b8-7450-4be2-915d2463f14e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d25b4eb-1708-bf1f-fb80-60924d330379 Address:8d25b4eb-1708-bf1f-fb80-60924d330379} {Suffrage:Voter ID:server-0b4e2022-61b8-7450-4be2-915d2463f14e Address:0b4e2022-61b8-7450-4be2-915d2463f14e} {Suffrage:Voter ID:server-25c69b24-8f21-961b-07e1-1ca51ad034ed Address:25c69b24-8f21-961b-07e1-1ca51ad034ed}]" +2024-07-05T13:36:26.985Z [INFO] server-0b4e2022-61b8-7450-4be2-915d2463f14e: entering follower state: follower="Node at 0b4e2022-61b8-7450-4be2-915d2463f14e [Follower]" leader-address= leader-id= +2024-07-05T13:36:26.986Z [INFO] server-25c69b24-8f21-961b-07e1-1ca51ad034ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d25b4eb-1708-bf1f-fb80-60924d330379 Address:8d25b4eb-1708-bf1f-fb80-60924d330379} {Suffrage:Voter ID:server-0b4e2022-61b8-7450-4be2-915d2463f14e Address:0b4e2022-61b8-7450-4be2-915d2463f14e} {Suffrage:Voter ID:server-25c69b24-8f21-961b-07e1-1ca51ad034ed Address:25c69b24-8f21-961b-07e1-1ca51ad034ed}]" +2024-07-05T13:36:26.986Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:26.986Z [INFO] server-25c69b24-8f21-961b-07e1-1ca51ad034ed: entering follower state: follower="Node at 25c69b24-8f21-961b-07e1-1ca51ad034ed [Follower]" leader-address= leader-id= +2024-07-05T13:36:27.657Z [WARN] server-25c69b24-8f21-961b-07e1-1ca51ad034ed: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:27.657Z [INFO] server-25c69b24-8f21-961b-07e1-1ca51ad034ed: entering candidate state: node="Node at 25c69b24-8f21-961b-07e1-1ca51ad034ed [Candidate]" term=2 +2024-07-05T13:36:27.658Z [INFO] server-25c69b24-8f21-961b-07e1-1ca51ad034ed: election won: term=2 tally=2 +2024-07-05T13:36:27.658Z [INFO] server-25c69b24-8f21-961b-07e1-1ca51ad034ed: entering leader state: leader="Node at 25c69b24-8f21-961b-07e1-1ca51ad034ed [Leader]" +2024-07-05T13:36:27.659Z [INFO] server-25c69b24-8f21-961b-07e1-1ca51ad034ed: added peer, starting replication: peer=server-8d25b4eb-1708-bf1f-fb80-60924d330379 +2024-07-05T13:36:27.659Z [INFO] server-25c69b24-8f21-961b-07e1-1ca51ad034ed: added peer, starting replication: peer=server-0b4e2022-61b8-7450-4be2-915d2463f14e +2024-07-05T13:36:27.660Z [INFO] server-25c69b24-8f21-961b-07e1-1ca51ad034ed: pipelining replication: peer="{Voter server-8d25b4eb-1708-bf1f-fb80-60924d330379 8d25b4eb-1708-bf1f-fb80-60924d330379}" +2024-07-05T13:36:27.660Z [INFO] server-25c69b24-8f21-961b-07e1-1ca51ad034ed: pipelining replication: peer="{Voter server-0b4e2022-61b8-7450-4be2-915d2463f14e 0b4e2022-61b8-7450-4be2-915d2463f14e}" +2024-07-05T13:36:28.664Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:27.659244748 +0000 UTC m=+53.655166491 (1 nodes), 672.477135ms from start of poll, 675.62416ms from cluster start. Timeout at 2024-07-05 13:36:28.664389931 +0000 UTC m=+54.660311091, 1.0051446s after stability +2024-07-05T13:36:28.670Z [INFO] server-25c69b24-8f21-961b-07e1-1ca51ad034ed: starting snapshot up to: index=12 +2024-07-05T13:36:28.670Z [INFO] creating new snapshot: path=/tmp/raft1169107925/snapshots/2-12-1720186588670.tmp +2024-07-05T13:36:28.674Z [INFO] server-25c69b24-8f21-961b-07e1-1ca51ad034ed: snapshot complete up to: index=12 +2024-07-05T13:36:28.686Z [INFO] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ed5de2b-5232-e749-d1f2-d5858a5ad980 Address:3ed5de2b-5232-e749-d1f2-d5858a5ad980} {Suffrage:Voter ID:server-d9c35f1c-6761-0615-0fda-43bad2334d4f Address:d9c35f1c-6761-0615-0fda-43bad2334d4f} {Suffrage:Voter ID:server-d0053970-d1d7-8ece-78f0-3f26aad1b4c1 Address:d0053970-d1d7-8ece-78f0-3f26aad1b4c1}]" +2024-07-05T13:36:28.686Z [INFO] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: entering follower state: follower="Node at 3ed5de2b-5232-e749-d1f2-d5858a5ad980 [Follower]" leader-address= leader-id= +2024-07-05T13:36:28.687Z [INFO] server-d9c35f1c-6761-0615-0fda-43bad2334d4f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ed5de2b-5232-e749-d1f2-d5858a5ad980 Address:3ed5de2b-5232-e749-d1f2-d5858a5ad980} {Suffrage:Voter ID:server-d9c35f1c-6761-0615-0fda-43bad2334d4f Address:d9c35f1c-6761-0615-0fda-43bad2334d4f} {Suffrage:Voter ID:server-d0053970-d1d7-8ece-78f0-3f26aad1b4c1 Address:d0053970-d1d7-8ece-78f0-3f26aad1b4c1}]" +2024-07-05T13:36:28.687Z [INFO] server-d9c35f1c-6761-0615-0fda-43bad2334d4f: entering follower state: follower="Node at d9c35f1c-6761-0615-0fda-43bad2334d4f [Follower]" leader-address= leader-id= +2024-07-05T13:36:28.688Z [INFO] server-d0053970-d1d7-8ece-78f0-3f26aad1b4c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ed5de2b-5232-e749-d1f2-d5858a5ad980 Address:3ed5de2b-5232-e749-d1f2-d5858a5ad980} {Suffrage:Voter ID:server-d9c35f1c-6761-0615-0fda-43bad2334d4f Address:d9c35f1c-6761-0615-0fda-43bad2334d4f} {Suffrage:Voter ID:server-d0053970-d1d7-8ece-78f0-3f26aad1b4c1 Address:d0053970-d1d7-8ece-78f0-3f26aad1b4c1}]" +2024-07-05T13:36:28.688Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:28.688Z [INFO] server-d0053970-d1d7-8ece-78f0-3f26aad1b4c1: entering follower state: follower="Node at d0053970-d1d7-8ece-78f0-3f26aad1b4c1 [Follower]" leader-address= leader-id= +2024-07-05T13:36:29.450Z [WARN] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:29.450Z [INFO] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: entering candidate state: node="Node at 3ed5de2b-5232-e749-d1f2-d5858a5ad980 [Candidate]" term=2 +2024-07-05T13:36:29.451Z [INFO] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: election won: term=2 tally=2 +2024-07-05T13:36:29.451Z [INFO] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: entering leader state: leader="Node at 3ed5de2b-5232-e749-d1f2-d5858a5ad980 [Leader]" +2024-07-05T13:36:29.451Z [INFO] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: added peer, starting replication: peer=server-d9c35f1c-6761-0615-0fda-43bad2334d4f +2024-07-05T13:36:29.452Z [INFO] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: added peer, starting replication: peer=server-d0053970-d1d7-8ece-78f0-3f26aad1b4c1 +2024-07-05T13:36:29.453Z [INFO] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: pipelining replication: peer="{Voter server-d9c35f1c-6761-0615-0fda-43bad2334d4f d9c35f1c-6761-0615-0fda-43bad2334d4f}" +2024-07-05T13:36:29.453Z [INFO] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: pipelining replication: peer="{Voter server-d0053970-d1d7-8ece-78f0-3f26aad1b4c1 d0053970-d1d7-8ece-78f0-3f26aad1b4c1}" +2024-07-05T13:36:30.459Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:29.452051129 +0000 UTC m=+55.447972581 (1 nodes), 763.642303ms from start of poll, 766.56825ms from cluster start. Timeout at 2024-07-05 13:36:30.458809783 +0000 UTC m=+56.454731527, 1.006758946s after stability +2024-07-05T13:36:30.460Z [INFO] creating new snapshot: path=/tmp/raft3479659824/snapshots/2-15-1720186590460.tmp +2024-07-05T13:36:30.465Z [INFO] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: copied to local snapshot: bytes=71 +2024-07-05T13:36:30.471Z [INFO] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: snapshot restore progress: id=2-15-1720186590460 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:30.472Z [INFO] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: removing all old logs from log store +2024-07-05T13:36:30.472Z [INFO] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: compacting logs: from=1 to=2 +2024-07-05T13:36:30.472Z [INFO] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: restored user snapshot: index=15 +2024-07-05T13:36:30.472Z [ERROR] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: failed to get log: index=2 error="log not found" +2024-07-05T13:36:30.473Z [INFO] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: aborting pipeline replication: peer="{Voter server-d0053970-d1d7-8ece-78f0-3f26aad1b4c1 d0053970-d1d7-8ece-78f0-3f26aad1b4c1}" +2024-07-05T13:36:30.479Z [ERROR] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: failed to get log: index=2 error="log not found" +2024-07-05T13:36:30.479Z [INFO] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: aborting pipeline replication: peer="{Voter server-d9c35f1c-6761-0615-0fda-43bad2334d4f d9c35f1c-6761-0615-0fda-43bad2334d4f}" +2024-07-05T13:36:30.479Z [ERROR] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: failed to get log: index=2 error="log not found" +2024-07-05T13:36:30.480Z [ERROR] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: failed to get log: index=2 error="log not found" +2024-07-05T13:36:30.482Z [INFO] creating new snapshot: path=/tmp/raft2329635862/snapshots/2-15-1720186590481.tmp +2024-07-05T13:36:30.482Z [INFO] creating new snapshot: path=/tmp/raft4052201155/snapshots/2-15-1720186590482.tmp +2024-07-05T13:36:30.484Z [INFO] server-d9c35f1c-6761-0615-0fda-43bad2334d4f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:30.484Z [INFO] server-d0053970-d1d7-8ece-78f0-3f26aad1b4c1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:30.486Z [INFO] server-d9c35f1c-6761-0615-0fda-43bad2334d4f: copied to local snapshot: bytes=71 +2024-07-05T13:36:30.487Z [INFO] server-d0053970-d1d7-8ece-78f0-3f26aad1b4c1: copied to local snapshot: bytes=71 +2024-07-05T13:36:30.488Z [INFO] server-d9c35f1c-6761-0615-0fda-43bad2334d4f: snapshot restore progress: id=2-15-1720186590481 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:30.488Z [INFO] server-d9c35f1c-6761-0615-0fda-43bad2334d4f: removing all old logs from log store +2024-07-05T13:36:30.488Z [INFO] server-d9c35f1c-6761-0615-0fda-43bad2334d4f: compacting logs: from=1 to=2 +2024-07-05T13:36:30.488Z [INFO] server-d9c35f1c-6761-0615-0fda-43bad2334d4f: Installed remote snapshot +2024-07-05T13:36:30.489Z [INFO] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: pipelining replication: peer="{Voter server-d9c35f1c-6761-0615-0fda-43bad2334d4f d9c35f1c-6761-0615-0fda-43bad2334d4f}" +2024-07-05T13:36:30.490Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:36:30.491Z [INFO] server-d0053970-d1d7-8ece-78f0-3f26aad1b4c1: snapshot restore progress: id=2-15-1720186590482 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:30.491Z [INFO] server-d0053970-d1d7-8ece-78f0-3f26aad1b4c1: removing all old logs from log store +2024-07-05T13:36:30.491Z [INFO] server-d0053970-d1d7-8ece-78f0-3f26aad1b4c1: compacting logs: from=1 to=2 +2024-07-05T13:36:30.491Z [INFO] server-d0053970-d1d7-8ece-78f0-3f26aad1b4c1: Installed remote snapshot +2024-07-05T13:36:30.492Z [INFO] server-3ed5de2b-5232-e749-d1f2-d5858a5ad980: pipelining replication: peer="{Voter server-d0053970-d1d7-8ece-78f0-3f26aad1b4c1 d0053970-d1d7-8ece-78f0-3f26aad1b4c1}" +2024-07-05T13:36:31.495Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:36:30.490314458 +0000 UTC m=+56.486235910 (2 nodes), 7.875µs from start of poll, 1.804831579s from cluster start. Timeout at 2024-07-05 13:36:31.495474517 +0000 UTC m=+57.491396260, 1.00516035s after stability +2024-07-05T13:36:31.511Z [INFO] server-25c69b24-8f21-961b-07e1-1ca51ad034ed: aborting pipeline replication: peer="{Voter server-0b4e2022-61b8-7450-4be2-915d2463f14e 0b4e2022-61b8-7450-4be2-915d2463f14e}" +2024-07-05T13:36:31.512Z [INFO] server-25c69b24-8f21-961b-07e1-1ca51ad034ed: aborting pipeline replication: peer="{Voter server-8d25b4eb-1708-bf1f-fb80-60924d330379 8d25b4eb-1708-bf1f-fb80-60924d330379}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false -2024-07-05T13:30:14.636Z [INFO] server-9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a Address:9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a} {Suffrage:Voter ID:server-119b8428-74b7-4fe1-489d-caf38a59ab8b Address:119b8428-74b7-4fe1-489d-caf38a59ab8b} {Suffrage:Voter ID:server-f9d9067f-481c-8f82-f02f-85cf0a775a5f Address:f9d9067f-481c-8f82-f02f-85cf0a775a5f}]" -2024-07-05T13:30:14.637Z [INFO] server-119b8428-74b7-4fe1-489d-caf38a59ab8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a Address:9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a} {Suffrage:Voter ID:server-119b8428-74b7-4fe1-489d-caf38a59ab8b Address:119b8428-74b7-4fe1-489d-caf38a59ab8b} {Suffrage:Voter ID:server-f9d9067f-481c-8f82-f02f-85cf0a775a5f Address:f9d9067f-481c-8f82-f02f-85cf0a775a5f}]" -2024-07-05T13:30:14.637Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a Address:9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a} {Suffrage:Voter ID:server-119b8428-74b7-4fe1-489d-caf38a59ab8b Address:119b8428-74b7-4fe1-489d-caf38a59ab8b} {Suffrage:Voter ID:server-f9d9067f-481c-8f82-f02f-85cf0a775a5f Address:f9d9067f-481c-8f82-f02f-85cf0a775a5f}]" -2024-07-05T13:30:14.637Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:14.637Z [INFO] server-9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a: entering follower state: follower="Node at 9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a [Follower]" leader-address= leader-id= -2024-07-05T13:30:14.637Z [INFO] server-119b8428-74b7-4fe1-489d-caf38a59ab8b: entering follower state: follower="Node at 119b8428-74b7-4fe1-489d-caf38a59ab8b [Follower]" leader-address= leader-id= -2024-07-05T13:30:14.637Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: entering follower state: follower="Node at f9d9067f-481c-8f82-f02f-85cf0a775a5f [Follower]" leader-address= leader-id= -2024-07-05T13:30:15.237Z [WARN] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:15.237Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: entering candidate state: node="Node at f9d9067f-481c-8f82-f02f-85cf0a775a5f [Candidate]" term=2 -2024-07-05T13:30:15.237Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: election won: term=2 tally=2 -2024-07-05T13:30:15.237Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: entering leader state: leader="Node at f9d9067f-481c-8f82-f02f-85cf0a775a5f [Leader]" -2024-07-05T13:30:15.237Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: added peer, starting replication: peer=server-9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a -2024-07-05T13:30:15.237Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: added peer, starting replication: peer=server-119b8428-74b7-4fe1-489d-caf38a59ab8b -2024-07-05T13:30:15.238Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: pipelining replication: peer="{Voter server-119b8428-74b7-4fe1-489d-caf38a59ab8b 119b8428-74b7-4fe1-489d-caf38a59ab8b}" -2024-07-05T13:30:15.238Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: pipelining replication: peer="{Voter server-9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a 9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a}" -2024-07-05T13:30:16.244Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:15.238532151 +0000 UTC m=+56.008442826 (1 nodes), 601.111151ms from start of poll, 601.73417ms from cluster start. Timeout at 2024-07-05 13:30:16.244143432 +0000 UTC m=+57.014054147, 1.005611321s after stability -2024-07-05T13:30:16.245Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: starting snapshot up to: index=12 -2024-07-05T13:30:16.245Z [INFO] creating new snapshot: path=/tmp/raft2699427332/snapshots/2-12-1720186216245.tmp -2024-07-05T13:30:16.247Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: snapshot complete up to: index=12 -2024-07-05T13:30:16.248Z [INFO] creating new snapshot: path=/tmp/raft2699427332/snapshots/2-113-1720186216248.tmp -2024-07-05T13:30:16.250Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: copied to local snapshot: bytes=71 -2024-07-05T13:30:16.250Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: snapshot restore progress: id=2-113-1720186216248 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:16.250Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: restored user snapshot: index=113 -2024-07-05T13:30:16.250Z [ERROR] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: failed to get log: index=23 error="log not found" -2024-07-05T13:30:16.250Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: aborting pipeline replication: peer="{Voter server-119b8428-74b7-4fe1-489d-caf38a59ab8b 119b8428-74b7-4fe1-489d-caf38a59ab8b}" -2024-07-05T13:30:16.250Z [ERROR] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: failed to get log: index=23 error="log not found" -2024-07-05T13:30:16.250Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: aborting pipeline replication: peer="{Voter server-9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a 9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a}" -2024-07-05T13:30:16.256Z [ERROR] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: failed to get log: index=23 error="log not found" -2024-07-05T13:30:16.256Z [ERROR] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: failed to get log: index=23 error="log not found" -2024-07-05T13:30:16.257Z [INFO] creating new snapshot: path=/tmp/raft2033112335/snapshots/2-113-1720186216256.tmp -2024-07-05T13:30:16.257Z [INFO] server-119b8428-74b7-4fe1-489d-caf38a59ab8b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:16.257Z [INFO] creating new snapshot: path=/tmp/raft3339531095/snapshots/2-113-1720186216257.tmp -2024-07-05T13:30:16.258Z [INFO] server-9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:16.258Z [INFO] server-119b8428-74b7-4fe1-489d-caf38a59ab8b: copied to local snapshot: bytes=71 -2024-07-05T13:30:16.259Z [INFO] server-119b8428-74b7-4fe1-489d-caf38a59ab8b: snapshot restore progress: id=2-113-1720186216256 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:16.259Z [INFO] server-119b8428-74b7-4fe1-489d-caf38a59ab8b: Installed remote snapshot -2024-07-05T13:30:16.259Z [WARN] server-119b8428-74b7-4fe1-489d-caf38a59ab8b: skipping application of old log: index=22 -2024-07-05T13:30:16.259Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: pipelining replication: peer="{Voter server-119b8428-74b7-4fe1-489d-caf38a59ab8b 119b8428-74b7-4fe1-489d-caf38a59ab8b}" -2024-07-05T13:30:16.259Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:16.259Z [INFO] server-9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a: copied to local snapshot: bytes=71 -2024-07-05T13:30:16.260Z [INFO] server-9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a: snapshot restore progress: id=2-113-1720186216257 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:16.260Z [INFO] server-9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a: Installed remote snapshot -2024-07-05T13:30:16.260Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: pipelining replication: peer="{Voter server-9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a 9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a}" -2024-07-05T13:30:17.265Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:16.259552411 +0000 UTC m=+57.029463086 (2 nodes), 1.48µs from start of poll, 1.62275443s from cluster start. Timeout at 2024-07-05 13:30:17.265266568 +0000 UTC m=+58.035177243, 1.005714157s after stability -2024-07-05T13:30:17.282Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: aborting pipeline replication: peer="{Voter server-119b8428-74b7-4fe1-489d-caf38a59ab8b 119b8428-74b7-4fe1-489d-caf38a59ab8b}" -2024-07-05T13:30:17.282Z [INFO] server-f9d9067f-481c-8f82-f02f-85cf0a775a5f: aborting pipeline replication: peer="{Voter server-9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a 9bbb3cd9-6a1d-e07d-6fa2-dd578a656a3a}" +2024-07-05T13:36:31.524Z [INFO] server-6d83c489-2358-d0c2-9d18-2334775522f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d83c489-2358-d0c2-9d18-2334775522f7 Address:6d83c489-2358-d0c2-9d18-2334775522f7} {Suffrage:Voter ID:server-b5c51113-2c50-231e-7183-74dd54a1ce8d Address:b5c51113-2c50-231e-7183-74dd54a1ce8d} {Suffrage:Voter ID:server-a3053824-680d-321e-a152-d967cd40dfdc Address:a3053824-680d-321e-a152-d967cd40dfdc}]" +2024-07-05T13:36:31.525Z [INFO] server-6d83c489-2358-d0c2-9d18-2334775522f7: entering follower state: follower="Node at 6d83c489-2358-d0c2-9d18-2334775522f7 [Follower]" leader-address= leader-id= +2024-07-05T13:36:31.525Z [INFO] server-b5c51113-2c50-231e-7183-74dd54a1ce8d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d83c489-2358-d0c2-9d18-2334775522f7 Address:6d83c489-2358-d0c2-9d18-2334775522f7} {Suffrage:Voter ID:server-b5c51113-2c50-231e-7183-74dd54a1ce8d Address:b5c51113-2c50-231e-7183-74dd54a1ce8d} {Suffrage:Voter ID:server-a3053824-680d-321e-a152-d967cd40dfdc Address:a3053824-680d-321e-a152-d967cd40dfdc}]" +2024-07-05T13:36:31.525Z [INFO] server-b5c51113-2c50-231e-7183-74dd54a1ce8d: entering follower state: follower="Node at b5c51113-2c50-231e-7183-74dd54a1ce8d [Follower]" leader-address= leader-id= +2024-07-05T13:36:31.526Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d83c489-2358-d0c2-9d18-2334775522f7 Address:6d83c489-2358-d0c2-9d18-2334775522f7} {Suffrage:Voter ID:server-b5c51113-2c50-231e-7183-74dd54a1ce8d Address:b5c51113-2c50-231e-7183-74dd54a1ce8d} {Suffrage:Voter ID:server-a3053824-680d-321e-a152-d967cd40dfdc Address:a3053824-680d-321e-a152-d967cd40dfdc}]" +2024-07-05T13:36:31.526Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:31.526Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: entering follower state: follower="Node at a3053824-680d-321e-a152-d967cd40dfdc [Follower]" leader-address= leader-id= +2024-07-05T13:36:32.082Z [WARN] server-a3053824-680d-321e-a152-d967cd40dfdc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:32.083Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: entering candidate state: node="Node at a3053824-680d-321e-a152-d967cd40dfdc [Candidate]" term=2 +2024-07-05T13:36:32.083Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: election won: term=2 tally=2 +2024-07-05T13:36:32.084Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: entering leader state: leader="Node at a3053824-680d-321e-a152-d967cd40dfdc [Leader]" +2024-07-05T13:36:32.084Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: added peer, starting replication: peer=server-6d83c489-2358-d0c2-9d18-2334775522f7 +2024-07-05T13:36:32.084Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: added peer, starting replication: peer=server-b5c51113-2c50-231e-7183-74dd54a1ce8d +2024-07-05T13:36:32.085Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: pipelining replication: peer="{Voter server-b5c51113-2c50-231e-7183-74dd54a1ce8d b5c51113-2c50-231e-7183-74dd54a1ce8d}" +2024-07-05T13:36:32.085Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: pipelining replication: peer="{Voter server-6d83c489-2358-d0c2-9d18-2334775522f7 6d83c489-2358-d0c2-9d18-2334775522f7}" +2024-07-05T13:36:33.089Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:32.084338762 +0000 UTC m=+58.080260213 (1 nodes), 557.333904ms from start of poll, 560.298642ms from cluster start. Timeout at 2024-07-05 13:36:33.089419197 +0000 UTC m=+59.085340649, 1.005080436s after stability +2024-07-05T13:36:33.094Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: starting snapshot up to: index=12 +2024-07-05T13:36:33.095Z [INFO] creating new snapshot: path=/tmp/raft3851885261/snapshots/2-12-1720186593095.tmp +2024-07-05T13:36:33.099Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: snapshot complete up to: index=12 +2024-07-05T13:36:33.105Z [INFO] creating new snapshot: path=/tmp/raft3851885261/snapshots/2-113-1720186593105.tmp +2024-07-05T13:36:33.109Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: copied to local snapshot: bytes=71 +2024-07-05T13:36:33.110Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: snapshot restore progress: id=2-113-1720186593105 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:33.111Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: restored user snapshot: index=113 +2024-07-05T13:36:33.111Z [ERROR] server-a3053824-680d-321e-a152-d967cd40dfdc: failed to get log: index=23 error="log not found" +2024-07-05T13:36:33.111Z [ERROR] server-a3053824-680d-321e-a152-d967cd40dfdc: failed to get log: index=23 error="log not found" +2024-07-05T13:36:33.111Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: aborting pipeline replication: peer="{Voter server-b5c51113-2c50-231e-7183-74dd54a1ce8d b5c51113-2c50-231e-7183-74dd54a1ce8d}" +2024-07-05T13:36:33.111Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: aborting pipeline replication: peer="{Voter server-6d83c489-2358-d0c2-9d18-2334775522f7 6d83c489-2358-d0c2-9d18-2334775522f7}" +2024-07-05T13:36:33.120Z [ERROR] server-a3053824-680d-321e-a152-d967cd40dfdc: failed to get log: index=23 error="log not found" +2024-07-05T13:36:33.120Z [ERROR] server-a3053824-680d-321e-a152-d967cd40dfdc: failed to get log: index=23 error="log not found" +2024-07-05T13:36:33.133Z [INFO] creating new snapshot: path=/tmp/raft1909169372/snapshots/2-113-1720186593133.tmp +2024-07-05T13:36:33.135Z [INFO] server-b5c51113-2c50-231e-7183-74dd54a1ce8d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:33.136Z [INFO] creating new snapshot: path=/tmp/raft1220022803/snapshots/2-113-1720186593136.tmp +2024-07-05T13:36:33.137Z [INFO] server-b5c51113-2c50-231e-7183-74dd54a1ce8d: copied to local snapshot: bytes=71 +2024-07-05T13:36:33.138Z [INFO] server-b5c51113-2c50-231e-7183-74dd54a1ce8d: snapshot restore progress: id=2-113-1720186593133 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:33.138Z [INFO] server-6d83c489-2358-d0c2-9d18-2334775522f7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:33.138Z [INFO] server-b5c51113-2c50-231e-7183-74dd54a1ce8d: Installed remote snapshot +2024-07-05T13:36:33.139Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: pipelining replication: peer="{Voter server-b5c51113-2c50-231e-7183-74dd54a1ce8d b5c51113-2c50-231e-7183-74dd54a1ce8d}" +2024-07-05T13:36:33.139Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:36:33.140Z [INFO] server-6d83c489-2358-d0c2-9d18-2334775522f7: copied to local snapshot: bytes=71 +2024-07-05T13:36:33.141Z [INFO] server-6d83c489-2358-d0c2-9d18-2334775522f7: snapshot restore progress: id=2-113-1720186593136 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:33.141Z [INFO] server-6d83c489-2358-d0c2-9d18-2334775522f7: Installed remote snapshot +2024-07-05T13:36:33.142Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: pipelining replication: peer="{Voter server-6d83c489-2358-d0c2-9d18-2334775522f7 6d83c489-2358-d0c2-9d18-2334775522f7}" +2024-07-05T13:36:34.145Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:36:33.139544406 +0000 UTC m=+59.135465275 (2 nodes), 5.542µs from start of poll, 1.615503704s from cluster start. Timeout at 2024-07-05 13:36:34.144770381 +0000 UTC m=+60.140692124, 1.005226849s after stability +2024-07-05T13:36:34.162Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: aborting pipeline replication: peer="{Voter server-b5c51113-2c50-231e-7183-74dd54a1ce8d b5c51113-2c50-231e-7183-74dd54a1ce8d}" +2024-07-05T13:36:34.162Z [INFO] server-a3053824-680d-321e-a152-d967cd40dfdc: aborting pipeline replication: peer="{Voter server-6d83c489-2358-d0c2-9d18-2334775522f7 6d83c489-2358-d0c2-9d18-2334775522f7}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false -2024-07-05T13:30:17.287Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8eb32c5-17d0-da6f-9f70-27c62bab834c Address:d8eb32c5-17d0-da6f-9f70-27c62bab834c} {Suffrage:Voter ID:server-ca6c0a4d-f747-4659-2bb0-2325f8355acb Address:ca6c0a4d-f747-4659-2bb0-2325f8355acb} {Suffrage:Voter ID:server-c149a0e0-bfbb-e23e-6f75-9203e3547f77 Address:c149a0e0-bfbb-e23e-6f75-9203e3547f77}]" -2024-07-05T13:30:17.287Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: entering follower state: follower="Node at d8eb32c5-17d0-da6f-9f70-27c62bab834c [Follower]" leader-address= leader-id= -2024-07-05T13:30:17.288Z [INFO] server-ca6c0a4d-f747-4659-2bb0-2325f8355acb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8eb32c5-17d0-da6f-9f70-27c62bab834c Address:d8eb32c5-17d0-da6f-9f70-27c62bab834c} {Suffrage:Voter ID:server-ca6c0a4d-f747-4659-2bb0-2325f8355acb Address:ca6c0a4d-f747-4659-2bb0-2325f8355acb} {Suffrage:Voter ID:server-c149a0e0-bfbb-e23e-6f75-9203e3547f77 Address:c149a0e0-bfbb-e23e-6f75-9203e3547f77}]" -2024-07-05T13:30:17.288Z [INFO] server-ca6c0a4d-f747-4659-2bb0-2325f8355acb: entering follower state: follower="Node at ca6c0a4d-f747-4659-2bb0-2325f8355acb [Follower]" leader-address= leader-id= -2024-07-05T13:30:17.288Z [INFO] server-c149a0e0-bfbb-e23e-6f75-9203e3547f77: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8eb32c5-17d0-da6f-9f70-27c62bab834c Address:d8eb32c5-17d0-da6f-9f70-27c62bab834c} {Suffrage:Voter ID:server-ca6c0a4d-f747-4659-2bb0-2325f8355acb Address:ca6c0a4d-f747-4659-2bb0-2325f8355acb} {Suffrage:Voter ID:server-c149a0e0-bfbb-e23e-6f75-9203e3547f77 Address:c149a0e0-bfbb-e23e-6f75-9203e3547f77}]" -2024-07-05T13:30:17.289Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:17.289Z [INFO] server-c149a0e0-bfbb-e23e-6f75-9203e3547f77: entering follower state: follower="Node at c149a0e0-bfbb-e23e-6f75-9203e3547f77 [Follower]" leader-address= leader-id= -2024-07-05T13:30:17.807Z [WARN] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:17.808Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: entering candidate state: node="Node at d8eb32c5-17d0-da6f-9f70-27c62bab834c [Candidate]" term=2 -2024-07-05T13:30:17.808Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: election won: term=2 tally=2 -2024-07-05T13:30:17.808Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: entering leader state: leader="Node at d8eb32c5-17d0-da6f-9f70-27c62bab834c [Leader]" -2024-07-05T13:30:17.808Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: added peer, starting replication: peer=server-ca6c0a4d-f747-4659-2bb0-2325f8355acb -2024-07-05T13:30:17.808Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: added peer, starting replication: peer=server-c149a0e0-bfbb-e23e-6f75-9203e3547f77 -2024-07-05T13:30:17.808Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: pipelining replication: peer="{Voter server-c149a0e0-bfbb-e23e-6f75-9203e3547f77 c149a0e0-bfbb-e23e-6f75-9203e3547f77}" -2024-07-05T13:30:17.808Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: pipelining replication: peer="{Voter server-ca6c0a4d-f747-4659-2bb0-2325f8355acb ca6c0a4d-f747-4659-2bb0-2325f8355acb}" -2024-07-05T13:30:18.815Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:17.808992932 +0000 UTC m=+58.578903607 (1 nodes), 519.91392ms from start of poll, 521.731696ms from cluster start. Timeout at 2024-07-05 13:30:18.814988159 +0000 UTC m=+59.584898834, 1.005995227s after stability -2024-07-05T13:30:18.816Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: starting snapshot up to: index=12 -2024-07-05T13:30:18.816Z [INFO] creating new snapshot: path=/tmp/raft3562272805/snapshots/2-12-1720186218816.tmp -2024-07-05T13:30:18.817Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: snapshot complete up to: index=12 -2024-07-05T13:30:18.819Z [INFO] creating new snapshot: path=/tmp/raft3562272805/snapshots/2-113-1720186218819.tmp -2024-07-05T13:30:18.822Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: copied to local snapshot: bytes=71 -2024-07-05T13:30:18.822Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: snapshot restore progress: id=2-113-1720186218819 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:18.822Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: removing all old logs from log store -2024-07-05T13:30:18.822Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: compacting logs: from=1 to=22 -2024-07-05T13:30:18.822Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: restored user snapshot: index=113 -2024-07-05T13:30:18.822Z [ERROR] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: failed to get log: index=22 error="log not found" -2024-07-05T13:30:18.822Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: aborting pipeline replication: peer="{Voter server-c149a0e0-bfbb-e23e-6f75-9203e3547f77 c149a0e0-bfbb-e23e-6f75-9203e3547f77}" -2024-07-05T13:30:18.823Z [ERROR] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: failed to get log: index=22 error="log not found" -2024-07-05T13:30:18.823Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: aborting pipeline replication: peer="{Voter server-ca6c0a4d-f747-4659-2bb0-2325f8355acb ca6c0a4d-f747-4659-2bb0-2325f8355acb}" -2024-07-05T13:30:18.832Z [ERROR] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: failed to get log: index=22 error="log not found" -2024-07-05T13:30:18.833Z [ERROR] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: failed to get log: index=22 error="log not found" -2024-07-05T13:30:18.833Z [INFO] creating new snapshot: path=/tmp/raft284976865/snapshots/2-113-1720186218833.tmp -2024-07-05T13:30:18.834Z [INFO] server-c149a0e0-bfbb-e23e-6f75-9203e3547f77: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:18.835Z [INFO] server-c149a0e0-bfbb-e23e-6f75-9203e3547f77: copied to local snapshot: bytes=71 -2024-07-05T13:30:18.835Z [INFO] server-c149a0e0-bfbb-e23e-6f75-9203e3547f77: snapshot restore progress: id=2-113-1720186218833 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:18.835Z [INFO] server-c149a0e0-bfbb-e23e-6f75-9203e3547f77: removing all old logs from log store -2024-07-05T13:30:18.835Z [INFO] server-c149a0e0-bfbb-e23e-6f75-9203e3547f77: compacting logs: from=1 to=22 -2024-07-05T13:30:18.835Z [INFO] server-c149a0e0-bfbb-e23e-6f75-9203e3547f77: Installed remote snapshot -2024-07-05T13:30:18.840Z [INFO] creating new snapshot: path=/tmp/raft1397750279/snapshots/2-113-1720186218840.tmp -2024-07-05T13:30:18.845Z [INFO] server-ca6c0a4d-f747-4659-2bb0-2325f8355acb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:18.846Z [INFO] server-ca6c0a4d-f747-4659-2bb0-2325f8355acb: copied to local snapshot: bytes=71 -2024-07-05T13:30:18.846Z [INFO] server-ca6c0a4d-f747-4659-2bb0-2325f8355acb: snapshot restore progress: id=2-113-1720186218840 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:18.846Z [INFO] server-ca6c0a4d-f747-4659-2bb0-2325f8355acb: removing all old logs from log store -2024-07-05T13:30:18.846Z [INFO] server-ca6c0a4d-f747-4659-2bb0-2325f8355acb: compacting logs: from=1 to=22 -2024-07-05T13:30:18.846Z [INFO] server-ca6c0a4d-f747-4659-2bb0-2325f8355acb: Installed remote snapshot -2024-07-05T13:30:18.846Z [WARN] server-ca6c0a4d-f747-4659-2bb0-2325f8355acb: skipping application of old log: index=22 -2024-07-05T13:30:18.847Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: pipelining replication: peer="{Voter server-ca6c0a4d-f747-4659-2bb0-2325f8355acb ca6c0a4d-f747-4659-2bb0-2325f8355acb}" -2024-07-05T13:30:18.847Z [WARN] server-c149a0e0-bfbb-e23e-6f75-9203e3547f77: skipping application of old log: index=22 -2024-07-05T13:30:18.847Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: pipelining replication: peer="{Voter server-c149a0e0-bfbb-e23e-6f75-9203e3547f77 c149a0e0-bfbb-e23e-6f75-9203e3547f77}" -2024-07-05T13:30:18.847Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:19.853Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:18.847375263 +0000 UTC m=+59.617285978 (2 nodes), 1.52µs from start of poll, 1.560114067s from cluster start. Timeout at 2024-07-05 13:30:19.853227335 +0000 UTC m=+60.623138010, 1.005852032s after stability -2024-07-05T13:30:19.875Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: aborting pipeline replication: peer="{Voter server-c149a0e0-bfbb-e23e-6f75-9203e3547f77 c149a0e0-bfbb-e23e-6f75-9203e3547f77}" -2024-07-05T13:30:19.875Z [INFO] server-d8eb32c5-17d0-da6f-9f70-27c62bab834c: aborting pipeline replication: peer="{Voter server-ca6c0a4d-f747-4659-2bb0-2325f8355acb ca6c0a4d-f747-4659-2bb0-2325f8355acb}" +2024-07-05T13:36:34.177Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fa39d26-513f-25ef-7ffb-b517733b414e Address:2fa39d26-513f-25ef-7ffb-b517733b414e} {Suffrage:Voter ID:server-cdc910e5-3907-a85b-1864-80c7f5698548 Address:cdc910e5-3907-a85b-1864-80c7f5698548} {Suffrage:Voter ID:server-67000869-d990-745f-6b1f-bef28494bad2 Address:67000869-d990-745f-6b1f-bef28494bad2}]" +2024-07-05T13:36:34.177Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: entering follower state: follower="Node at 2fa39d26-513f-25ef-7ffb-b517733b414e [Follower]" leader-address= leader-id= +2024-07-05T13:36:34.178Z [INFO] server-cdc910e5-3907-a85b-1864-80c7f5698548: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fa39d26-513f-25ef-7ffb-b517733b414e Address:2fa39d26-513f-25ef-7ffb-b517733b414e} {Suffrage:Voter ID:server-cdc910e5-3907-a85b-1864-80c7f5698548 Address:cdc910e5-3907-a85b-1864-80c7f5698548} {Suffrage:Voter ID:server-67000869-d990-745f-6b1f-bef28494bad2 Address:67000869-d990-745f-6b1f-bef28494bad2}]" +2024-07-05T13:36:34.178Z [INFO] server-cdc910e5-3907-a85b-1864-80c7f5698548: entering follower state: follower="Node at cdc910e5-3907-a85b-1864-80c7f5698548 [Follower]" leader-address= leader-id= +2024-07-05T13:36:34.179Z [INFO] server-67000869-d990-745f-6b1f-bef28494bad2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fa39d26-513f-25ef-7ffb-b517733b414e Address:2fa39d26-513f-25ef-7ffb-b517733b414e} {Suffrage:Voter ID:server-cdc910e5-3907-a85b-1864-80c7f5698548 Address:cdc910e5-3907-a85b-1864-80c7f5698548} {Suffrage:Voter ID:server-67000869-d990-745f-6b1f-bef28494bad2 Address:67000869-d990-745f-6b1f-bef28494bad2}]" +2024-07-05T13:36:34.179Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:34.179Z [INFO] server-67000869-d990-745f-6b1f-bef28494bad2: entering follower state: follower="Node at 67000869-d990-745f-6b1f-bef28494bad2 [Follower]" leader-address= leader-id= +2024-07-05T13:36:34.732Z [WARN] server-2fa39d26-513f-25ef-7ffb-b517733b414e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:34.732Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: entering candidate state: node="Node at 2fa39d26-513f-25ef-7ffb-b517733b414e [Candidate]" term=2 +2024-07-05T13:36:34.733Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: election won: term=2 tally=2 +2024-07-05T13:36:34.733Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: entering leader state: leader="Node at 2fa39d26-513f-25ef-7ffb-b517733b414e [Leader]" +2024-07-05T13:36:34.733Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: added peer, starting replication: peer=server-cdc910e5-3907-a85b-1864-80c7f5698548 +2024-07-05T13:36:34.733Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: added peer, starting replication: peer=server-67000869-d990-745f-6b1f-bef28494bad2 +2024-07-05T13:36:34.734Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: pipelining replication: peer="{Voter server-cdc910e5-3907-a85b-1864-80c7f5698548 cdc910e5-3907-a85b-1864-80c7f5698548}" +2024-07-05T13:36:34.734Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: pipelining replication: peer="{Voter server-67000869-d990-745f-6b1f-bef28494bad2 67000869-d990-745f-6b1f-bef28494bad2}" +2024-07-05T13:36:35.738Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:34.733463129 +0000 UTC m=+60.729384581 (1 nodes), 553.717012ms from start of poll, 556.729874ms from cluster start. Timeout at 2024-07-05 13:36:35.738475607 +0000 UTC m=+61.734397059, 1.005012478s after stability +2024-07-05T13:36:35.744Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: starting snapshot up to: index=12 +2024-07-05T13:36:35.744Z [INFO] creating new snapshot: path=/tmp/raft1144421708/snapshots/2-12-1720186595744.tmp +2024-07-05T13:36:35.753Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: snapshot complete up to: index=12 +2024-07-05T13:36:35.770Z [INFO] creating new snapshot: path=/tmp/raft1144421708/snapshots/2-113-1720186595770.tmp +2024-07-05T13:36:35.773Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: copied to local snapshot: bytes=71 +2024-07-05T13:36:35.774Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: snapshot restore progress: id=2-113-1720186595770 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:35.775Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: removing all old logs from log store +2024-07-05T13:36:35.775Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: compacting logs: from=1 to=22 +2024-07-05T13:36:35.775Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: restored user snapshot: index=113 +2024-07-05T13:36:35.775Z [ERROR] server-2fa39d26-513f-25ef-7ffb-b517733b414e: failed to get log: index=22 error="log not found" +2024-07-05T13:36:35.775Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: aborting pipeline replication: peer="{Voter server-cdc910e5-3907-a85b-1864-80c7f5698548 cdc910e5-3907-a85b-1864-80c7f5698548}" +2024-07-05T13:36:35.775Z [ERROR] server-2fa39d26-513f-25ef-7ffb-b517733b414e: failed to get log: index=22 error="log not found" +2024-07-05T13:36:35.775Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: aborting pipeline replication: peer="{Voter server-67000869-d990-745f-6b1f-bef28494bad2 67000869-d990-745f-6b1f-bef28494bad2}" +2024-07-05T13:36:35.781Z [ERROR] server-2fa39d26-513f-25ef-7ffb-b517733b414e: failed to get log: index=22 error="log not found" +2024-07-05T13:36:35.784Z [INFO] creating new snapshot: path=/tmp/raft1277078607/snapshots/2-113-1720186595784.tmp +2024-07-05T13:36:35.785Z [ERROR] server-2fa39d26-513f-25ef-7ffb-b517733b414e: failed to get log: index=22 error="log not found" +2024-07-05T13:36:35.786Z [INFO] server-cdc910e5-3907-a85b-1864-80c7f5698548: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:35.788Z [INFO] creating new snapshot: path=/tmp/raft3496524396/snapshots/2-113-1720186595788.tmp +2024-07-05T13:36:35.788Z [INFO] server-cdc910e5-3907-a85b-1864-80c7f5698548: copied to local snapshot: bytes=71 +2024-07-05T13:36:35.789Z [INFO] server-cdc910e5-3907-a85b-1864-80c7f5698548: snapshot restore progress: id=2-113-1720186595784 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:35.789Z [INFO] server-cdc910e5-3907-a85b-1864-80c7f5698548: removing all old logs from log store +2024-07-05T13:36:35.789Z [INFO] server-cdc910e5-3907-a85b-1864-80c7f5698548: compacting logs: from=1 to=22 +2024-07-05T13:36:35.789Z [INFO] server-cdc910e5-3907-a85b-1864-80c7f5698548: Installed remote snapshot +2024-07-05T13:36:35.789Z [INFO] server-67000869-d990-745f-6b1f-bef28494bad2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:35.790Z [WARN] server-cdc910e5-3907-a85b-1864-80c7f5698548: skipping application of old log: index=22 +2024-07-05T13:36:35.790Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: pipelining replication: peer="{Voter server-cdc910e5-3907-a85b-1864-80c7f5698548 cdc910e5-3907-a85b-1864-80c7f5698548}" +2024-07-05T13:36:35.790Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:36:35.791Z [INFO] server-67000869-d990-745f-6b1f-bef28494bad2: copied to local snapshot: bytes=71 +2024-07-05T13:36:35.793Z [INFO] server-67000869-d990-745f-6b1f-bef28494bad2: snapshot restore progress: id=2-113-1720186595788 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:35.793Z [INFO] server-67000869-d990-745f-6b1f-bef28494bad2: removing all old logs from log store +2024-07-05T13:36:35.793Z [INFO] server-67000869-d990-745f-6b1f-bef28494bad2: compacting logs: from=1 to=22 +2024-07-05T13:36:35.793Z [INFO] server-67000869-d990-745f-6b1f-bef28494bad2: Installed remote snapshot +2024-07-05T13:36:35.794Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: pipelining replication: peer="{Voter server-67000869-d990-745f-6b1f-bef28494bad2 67000869-d990-745f-6b1f-bef28494bad2}" +2024-07-05T13:36:36.796Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:36:35.790765236 +0000 UTC m=+61.786686396 (2 nodes), 5.834µs from start of poll, 1.614031689s from cluster start. Timeout at 2024-07-05 13:36:36.795796089 +0000 UTC m=+62.791717540, 1.005031144s after stability +2024-07-05T13:36:36.825Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: aborting pipeline replication: peer="{Voter server-67000869-d990-745f-6b1f-bef28494bad2 67000869-d990-745f-6b1f-bef28494bad2}" +2024-07-05T13:36:36.825Z [INFO] server-2fa39d26-513f-25ef-7ffb-b517733b414e: aborting pipeline replication: peer="{Voter server-cdc910e5-3907-a85b-1864-80c7f5698548 cdc910e5-3907-a85b-1864-80c7f5698548}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true -2024-07-05T13:30:19.878Z [INFO] server-37c1b440-13a4-cba7-b049-6a5a518efed8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37c1b440-13a4-cba7-b049-6a5a518efed8 Address:37c1b440-13a4-cba7-b049-6a5a518efed8} {Suffrage:Voter ID:server-078f0d95-337f-0de2-a127-538f1462586d Address:078f0d95-337f-0de2-a127-538f1462586d} {Suffrage:Voter ID:server-8fd99096-2642-02d0-8d88-87e4043d7c68 Address:8fd99096-2642-02d0-8d88-87e4043d7c68}]" -2024-07-05T13:30:19.878Z [INFO] server-078f0d95-337f-0de2-a127-538f1462586d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37c1b440-13a4-cba7-b049-6a5a518efed8 Address:37c1b440-13a4-cba7-b049-6a5a518efed8} {Suffrage:Voter ID:server-078f0d95-337f-0de2-a127-538f1462586d Address:078f0d95-337f-0de2-a127-538f1462586d} {Suffrage:Voter ID:server-8fd99096-2642-02d0-8d88-87e4043d7c68 Address:8fd99096-2642-02d0-8d88-87e4043d7c68}]" -2024-07-05T13:30:19.878Z [INFO] server-8fd99096-2642-02d0-8d88-87e4043d7c68: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37c1b440-13a4-cba7-b049-6a5a518efed8 Address:37c1b440-13a4-cba7-b049-6a5a518efed8} {Suffrage:Voter ID:server-078f0d95-337f-0de2-a127-538f1462586d Address:078f0d95-337f-0de2-a127-538f1462586d} {Suffrage:Voter ID:server-8fd99096-2642-02d0-8d88-87e4043d7c68 Address:8fd99096-2642-02d0-8d88-87e4043d7c68}]" -2024-07-05T13:30:19.879Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:19.879Z [INFO] server-37c1b440-13a4-cba7-b049-6a5a518efed8: entering follower state: follower="Node at 37c1b440-13a4-cba7-b049-6a5a518efed8 [Follower]" leader-address= leader-id= -2024-07-05T13:30:19.879Z [INFO] server-078f0d95-337f-0de2-a127-538f1462586d: entering follower state: follower="Node at 078f0d95-337f-0de2-a127-538f1462586d [Follower]" leader-address= leader-id= -2024-07-05T13:30:19.879Z [INFO] server-8fd99096-2642-02d0-8d88-87e4043d7c68: entering follower state: follower="Node at 8fd99096-2642-02d0-8d88-87e4043d7c68 [Follower]" leader-address= leader-id= -2024-07-05T13:30:20.741Z [WARN] server-37c1b440-13a4-cba7-b049-6a5a518efed8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:20.741Z [INFO] server-37c1b440-13a4-cba7-b049-6a5a518efed8: entering candidate state: node="Node at 37c1b440-13a4-cba7-b049-6a5a518efed8 [Candidate]" term=2 -2024-07-05T13:30:20.741Z [INFO] server-37c1b440-13a4-cba7-b049-6a5a518efed8: election won: term=2 tally=2 -2024-07-05T13:30:20.741Z [INFO] server-37c1b440-13a4-cba7-b049-6a5a518efed8: entering leader state: leader="Node at 37c1b440-13a4-cba7-b049-6a5a518efed8 [Leader]" -2024-07-05T13:30:20.741Z [INFO] server-37c1b440-13a4-cba7-b049-6a5a518efed8: added peer, starting replication: peer=server-078f0d95-337f-0de2-a127-538f1462586d -2024-07-05T13:30:20.741Z [INFO] server-37c1b440-13a4-cba7-b049-6a5a518efed8: added peer, starting replication: peer=server-8fd99096-2642-02d0-8d88-87e4043d7c68 -2024-07-05T13:30:20.742Z [INFO] server-37c1b440-13a4-cba7-b049-6a5a518efed8: pipelining replication: peer="{Voter server-078f0d95-337f-0de2-a127-538f1462586d 078f0d95-337f-0de2-a127-538f1462586d}" -2024-07-05T13:30:20.742Z [INFO] server-37c1b440-13a4-cba7-b049-6a5a518efed8: pipelining replication: peer="{Voter server-8fd99096-2642-02d0-8d88-87e4043d7c68 8fd99096-2642-02d0-8d88-87e4043d7c68}" -2024-07-05T13:30:21.747Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:20.741873839 +0000 UTC m=+61.511784514 (1 nodes), 862.838329ms from start of poll, 863.532505ms from cluster start. Timeout at 2024-07-05 13:30:21.747679272 +0000 UTC m=+62.517590027, 1.005805513s after stability -2024-07-05T13:30:21.749Z [INFO] server-37c1b440-13a4-cba7-b049-6a5a518efed8: starting snapshot up to: index=12 -2024-07-05T13:30:21.749Z [INFO] creating new snapshot: path=/tmp/raft614296819/snapshots/2-12-1720186221749.tmp -2024-07-05T13:30:21.750Z [INFO] server-37c1b440-13a4-cba7-b049-6a5a518efed8: snapshot complete up to: index=12 -2024-07-05T13:30:21.754Z [INFO] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f8149a2-cb27-390f-95c4-26e2deca8edb Address:2f8149a2-cb27-390f-95c4-26e2deca8edb} {Suffrage:Voter ID:server-3c4a4685-e5d4-2e61-2408-4d4cf8addc03 Address:3c4a4685-e5d4-2e61-2408-4d4cf8addc03} {Suffrage:Voter ID:server-6f458dc0-482a-7b7f-a633-382041503967 Address:6f458dc0-482a-7b7f-a633-382041503967}]" -2024-07-05T13:30:21.754Z [INFO] server-3c4a4685-e5d4-2e61-2408-4d4cf8addc03: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f8149a2-cb27-390f-95c4-26e2deca8edb Address:2f8149a2-cb27-390f-95c4-26e2deca8edb} {Suffrage:Voter ID:server-3c4a4685-e5d4-2e61-2408-4d4cf8addc03 Address:3c4a4685-e5d4-2e61-2408-4d4cf8addc03} {Suffrage:Voter ID:server-6f458dc0-482a-7b7f-a633-382041503967 Address:6f458dc0-482a-7b7f-a633-382041503967}]" -2024-07-05T13:30:21.755Z [INFO] server-6f458dc0-482a-7b7f-a633-382041503967: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f8149a2-cb27-390f-95c4-26e2deca8edb Address:2f8149a2-cb27-390f-95c4-26e2deca8edb} {Suffrage:Voter ID:server-3c4a4685-e5d4-2e61-2408-4d4cf8addc03 Address:3c4a4685-e5d4-2e61-2408-4d4cf8addc03} {Suffrage:Voter ID:server-6f458dc0-482a-7b7f-a633-382041503967 Address:6f458dc0-482a-7b7f-a633-382041503967}]" -2024-07-05T13:30:21.755Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:21.755Z [INFO] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: entering follower state: follower="Node at 2f8149a2-cb27-390f-95c4-26e2deca8edb [Follower]" leader-address= leader-id= -2024-07-05T13:30:21.755Z [INFO] server-3c4a4685-e5d4-2e61-2408-4d4cf8addc03: entering follower state: follower="Node at 3c4a4685-e5d4-2e61-2408-4d4cf8addc03 [Follower]" leader-address= leader-id= -2024-07-05T13:30:21.755Z [INFO] server-6f458dc0-482a-7b7f-a633-382041503967: entering follower state: follower="Node at 6f458dc0-482a-7b7f-a633-382041503967 [Follower]" leader-address= leader-id= -2024-07-05T13:30:22.367Z [WARN] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:22.367Z [INFO] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: entering candidate state: node="Node at 2f8149a2-cb27-390f-95c4-26e2deca8edb [Candidate]" term=2 -2024-07-05T13:30:22.367Z [INFO] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: election won: term=2 tally=2 -2024-07-05T13:30:22.367Z [INFO] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: entering leader state: leader="Node at 2f8149a2-cb27-390f-95c4-26e2deca8edb [Leader]" -2024-07-05T13:30:22.367Z [INFO] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: added peer, starting replication: peer=server-3c4a4685-e5d4-2e61-2408-4d4cf8addc03 -2024-07-05T13:30:22.367Z [INFO] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: added peer, starting replication: peer=server-6f458dc0-482a-7b7f-a633-382041503967 -2024-07-05T13:30:22.367Z [WARN] server-6f458dc0-482a-7b7f-a633-382041503967: rejecting vote request since our last term is greater: candidate=2f8149a2-cb27-390f-95c4-26e2deca8edb last-term=2 last-candidate-term=1 -2024-07-05T13:30:22.368Z [INFO] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: pipelining replication: peer="{Voter server-3c4a4685-e5d4-2e61-2408-4d4cf8addc03 3c4a4685-e5d4-2e61-2408-4d4cf8addc03}" -2024-07-05T13:30:22.368Z [INFO] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: pipelining replication: peer="{Voter server-6f458dc0-482a-7b7f-a633-382041503967 6f458dc0-482a-7b7f-a633-382041503967}" -2024-07-05T13:30:23.376Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:22.367966791 +0000 UTC m=+63.137877466 (1 nodes), 612.696185ms from start of poll, 613.472398ms from cluster start. Timeout at 2024-07-05 13:30:23.376768959 +0000 UTC m=+64.146679634, 1.008802168s after stability -2024-07-05T13:30:23.377Z [INFO] creating new snapshot: path=/tmp/raft1912260912/snapshots/2-113-1720186223377.tmp -2024-07-05T13:30:23.378Z [INFO] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: copied to local snapshot: bytes=71 -2024-07-05T13:30:23.379Z [INFO] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: snapshot restore progress: id=2-113-1720186223377 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:23.379Z [INFO] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: restored user snapshot: index=113 -2024-07-05T13:30:23.379Z [ERROR] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: failed to get log: index=3 error="log not found" -2024-07-05T13:30:23.379Z [INFO] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: aborting pipeline replication: peer="{Voter server-6f458dc0-482a-7b7f-a633-382041503967 6f458dc0-482a-7b7f-a633-382041503967}" -2024-07-05T13:30:23.379Z [ERROR] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: failed to get log: index=3 error="log not found" -2024-07-05T13:30:23.379Z [INFO] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: aborting pipeline replication: peer="{Voter server-3c4a4685-e5d4-2e61-2408-4d4cf8addc03 3c4a4685-e5d4-2e61-2408-4d4cf8addc03}" -2024-07-05T13:30:23.386Z [ERROR] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: failed to get log: index=3 error="log not found" -2024-07-05T13:30:23.387Z [INFO] creating new snapshot: path=/tmp/raft3707192000/snapshots/2-113-1720186223387.tmp -2024-07-05T13:30:23.387Z [INFO] server-6f458dc0-482a-7b7f-a633-382041503967: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:23.388Z [INFO] server-6f458dc0-482a-7b7f-a633-382041503967: copied to local snapshot: bytes=71 -2024-07-05T13:30:23.389Z [ERROR] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: failed to get log: index=3 error="log not found" -2024-07-05T13:30:23.395Z [INFO] creating new snapshot: path=/tmp/raft938861757/snapshots/2-113-1720186223395.tmp -2024-07-05T13:30:23.397Z [INFO] server-3c4a4685-e5d4-2e61-2408-4d4cf8addc03: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:23.399Z [INFO] server-3c4a4685-e5d4-2e61-2408-4d4cf8addc03: copied to local snapshot: bytes=71 -2024-07-05T13:30:23.399Z [INFO] server-6f458dc0-482a-7b7f-a633-382041503967: snapshot restore progress: id=2-113-1720186223387 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:23.399Z [INFO] server-6f458dc0-482a-7b7f-a633-382041503967: Installed remote snapshot -2024-07-05T13:30:23.400Z [INFO] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: pipelining replication: peer="{Voter server-6f458dc0-482a-7b7f-a633-382041503967 6f458dc0-482a-7b7f-a633-382041503967}" -2024-07-05T13:30:23.400Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:23.401Z [INFO] server-3c4a4685-e5d4-2e61-2408-4d4cf8addc03: snapshot restore progress: id=2-113-1720186223395 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:23.401Z [INFO] server-3c4a4685-e5d4-2e61-2408-4d4cf8addc03: Installed remote snapshot -2024-07-05T13:30:23.401Z [INFO] server-2f8149a2-cb27-390f-95c4-26e2deca8edb: pipelining replication: peer="{Voter server-3c4a4685-e5d4-2e61-2408-4d4cf8addc03 3c4a4685-e5d4-2e61-2408-4d4cf8addc03}" -2024-07-05T13:30:24.405Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:23.400377451 +0000 UTC m=+64.170288126 (2 nodes), 2.2µs from start of poll, 1.645883058s from cluster start. Timeout at 2024-07-05 13:30:24.405418231 +0000 UTC m=+65.175328906, 1.00504078s after stability -2024-07-05T13:30:24.418Z [INFO] server-37c1b440-13a4-cba7-b049-6a5a518efed8: aborting pipeline replication: peer="{Voter server-8fd99096-2642-02d0-8d88-87e4043d7c68 8fd99096-2642-02d0-8d88-87e4043d7c68}" -2024-07-05T13:30:24.418Z [INFO] server-37c1b440-13a4-cba7-b049-6a5a518efed8: aborting pipeline replication: peer="{Voter server-078f0d95-337f-0de2-a127-538f1462586d 078f0d95-337f-0de2-a127-538f1462586d}" +2024-07-05T13:36:36.839Z [INFO] server-a6986aa4-4362-0bd5-a187-93f91f2e2cd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6986aa4-4362-0bd5-a187-93f91f2e2cd9 Address:a6986aa4-4362-0bd5-a187-93f91f2e2cd9} {Suffrage:Voter ID:server-1be08a39-07e5-223c-70f3-5cf9f70d618d Address:1be08a39-07e5-223c-70f3-5cf9f70d618d} {Suffrage:Voter ID:server-89c91cb1-ee69-cb65-e0f5-65aa26ad15ff Address:89c91cb1-ee69-cb65-e0f5-65aa26ad15ff}]" +2024-07-05T13:36:36.839Z [INFO] server-a6986aa4-4362-0bd5-a187-93f91f2e2cd9: entering follower state: follower="Node at a6986aa4-4362-0bd5-a187-93f91f2e2cd9 [Follower]" leader-address= leader-id= +2024-07-05T13:36:36.840Z [INFO] server-1be08a39-07e5-223c-70f3-5cf9f70d618d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6986aa4-4362-0bd5-a187-93f91f2e2cd9 Address:a6986aa4-4362-0bd5-a187-93f91f2e2cd9} {Suffrage:Voter ID:server-1be08a39-07e5-223c-70f3-5cf9f70d618d Address:1be08a39-07e5-223c-70f3-5cf9f70d618d} {Suffrage:Voter ID:server-89c91cb1-ee69-cb65-e0f5-65aa26ad15ff Address:89c91cb1-ee69-cb65-e0f5-65aa26ad15ff}]" +2024-07-05T13:36:36.840Z [INFO] server-1be08a39-07e5-223c-70f3-5cf9f70d618d: entering follower state: follower="Node at 1be08a39-07e5-223c-70f3-5cf9f70d618d [Follower]" leader-address= leader-id= +2024-07-05T13:36:36.841Z [INFO] server-89c91cb1-ee69-cb65-e0f5-65aa26ad15ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6986aa4-4362-0bd5-a187-93f91f2e2cd9 Address:a6986aa4-4362-0bd5-a187-93f91f2e2cd9} {Suffrage:Voter ID:server-1be08a39-07e5-223c-70f3-5cf9f70d618d Address:1be08a39-07e5-223c-70f3-5cf9f70d618d} {Suffrage:Voter ID:server-89c91cb1-ee69-cb65-e0f5-65aa26ad15ff Address:89c91cb1-ee69-cb65-e0f5-65aa26ad15ff}]" +2024-07-05T13:36:36.841Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:36.841Z [INFO] server-89c91cb1-ee69-cb65-e0f5-65aa26ad15ff: entering follower state: follower="Node at 89c91cb1-ee69-cb65-e0f5-65aa26ad15ff [Follower]" leader-address= leader-id= +2024-07-05T13:36:37.581Z [WARN] server-a6986aa4-4362-0bd5-a187-93f91f2e2cd9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:37.581Z [INFO] server-a6986aa4-4362-0bd5-a187-93f91f2e2cd9: entering candidate state: node="Node at a6986aa4-4362-0bd5-a187-93f91f2e2cd9 [Candidate]" term=2 +2024-07-05T13:36:37.582Z [INFO] server-a6986aa4-4362-0bd5-a187-93f91f2e2cd9: election won: term=2 tally=2 +2024-07-05T13:36:37.582Z [INFO] server-a6986aa4-4362-0bd5-a187-93f91f2e2cd9: entering leader state: leader="Node at a6986aa4-4362-0bd5-a187-93f91f2e2cd9 [Leader]" +2024-07-05T13:36:37.582Z [INFO] server-a6986aa4-4362-0bd5-a187-93f91f2e2cd9: added peer, starting replication: peer=server-1be08a39-07e5-223c-70f3-5cf9f70d618d +2024-07-05T13:36:37.582Z [INFO] server-a6986aa4-4362-0bd5-a187-93f91f2e2cd9: added peer, starting replication: peer=server-89c91cb1-ee69-cb65-e0f5-65aa26ad15ff +2024-07-05T13:36:37.583Z [INFO] server-a6986aa4-4362-0bd5-a187-93f91f2e2cd9: pipelining replication: peer="{Voter server-89c91cb1-ee69-cb65-e0f5-65aa26ad15ff 89c91cb1-ee69-cb65-e0f5-65aa26ad15ff}" +2024-07-05T13:36:37.584Z [INFO] server-a6986aa4-4362-0bd5-a187-93f91f2e2cd9: pipelining replication: peer="{Voter server-1be08a39-07e5-223c-70f3-5cf9f70d618d 1be08a39-07e5-223c-70f3-5cf9f70d618d}" +2024-07-05T13:36:38.589Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:37.583185167 +0000 UTC m=+63.579106618 (1 nodes), 741.775286ms from start of poll, 744.544027ms from cluster start. Timeout at 2024-07-05 13:36:38.588761427 +0000 UTC m=+64.584682878, 1.00557626s after stability +2024-07-05T13:36:38.594Z [INFO] server-a6986aa4-4362-0bd5-a187-93f91f2e2cd9: starting snapshot up to: index=12 +2024-07-05T13:36:38.594Z [INFO] creating new snapshot: path=/tmp/raft4101827775/snapshots/2-12-1720186598594.tmp +2024-07-05T13:36:38.598Z [INFO] server-a6986aa4-4362-0bd5-a187-93f91f2e2cd9: snapshot complete up to: index=12 +2024-07-05T13:36:38.610Z [INFO] server-2d6d9d6c-03ad-3458-4d04-3321d2205e39: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d6d9d6c-03ad-3458-4d04-3321d2205e39 Address:2d6d9d6c-03ad-3458-4d04-3321d2205e39} {Suffrage:Voter ID:server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0 Address:0b43d00d-8fbd-a589-59ec-64dfb5c86ec0} {Suffrage:Voter ID:server-8308c6e0-dfce-43e3-3760-af609d174410 Address:8308c6e0-dfce-43e3-3760-af609d174410}]" +2024-07-05T13:36:38.611Z [INFO] server-2d6d9d6c-03ad-3458-4d04-3321d2205e39: entering follower state: follower="Node at 2d6d9d6c-03ad-3458-4d04-3321d2205e39 [Follower]" leader-address= leader-id= +2024-07-05T13:36:38.611Z [INFO] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d6d9d6c-03ad-3458-4d04-3321d2205e39 Address:2d6d9d6c-03ad-3458-4d04-3321d2205e39} {Suffrage:Voter ID:server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0 Address:0b43d00d-8fbd-a589-59ec-64dfb5c86ec0} {Suffrage:Voter ID:server-8308c6e0-dfce-43e3-3760-af609d174410 Address:8308c6e0-dfce-43e3-3760-af609d174410}]" +2024-07-05T13:36:38.611Z [INFO] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: entering follower state: follower="Node at 0b43d00d-8fbd-a589-59ec-64dfb5c86ec0 [Follower]" leader-address= leader-id= +2024-07-05T13:36:38.612Z [INFO] server-8308c6e0-dfce-43e3-3760-af609d174410: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d6d9d6c-03ad-3458-4d04-3321d2205e39 Address:2d6d9d6c-03ad-3458-4d04-3321d2205e39} {Suffrage:Voter ID:server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0 Address:0b43d00d-8fbd-a589-59ec-64dfb5c86ec0} {Suffrage:Voter ID:server-8308c6e0-dfce-43e3-3760-af609d174410 Address:8308c6e0-dfce-43e3-3760-af609d174410}]" +2024-07-05T13:36:38.612Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:38.613Z [INFO] server-8308c6e0-dfce-43e3-3760-af609d174410: entering follower state: follower="Node at 8308c6e0-dfce-43e3-3760-af609d174410 [Follower]" leader-address= leader-id= +2024-07-05T13:36:39.281Z [WARN] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:39.281Z [INFO] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: entering candidate state: node="Node at 0b43d00d-8fbd-a589-59ec-64dfb5c86ec0 [Candidate]" term=2 +2024-07-05T13:36:39.282Z [INFO] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: election won: term=2 tally=2 +2024-07-05T13:36:39.282Z [INFO] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: entering leader state: leader="Node at 0b43d00d-8fbd-a589-59ec-64dfb5c86ec0 [Leader]" +2024-07-05T13:36:39.282Z [INFO] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: added peer, starting replication: peer=server-2d6d9d6c-03ad-3458-4d04-3321d2205e39 +2024-07-05T13:36:39.282Z [INFO] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: added peer, starting replication: peer=server-8308c6e0-dfce-43e3-3760-af609d174410 +2024-07-05T13:36:39.283Z [INFO] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: pipelining replication: peer="{Voter server-2d6d9d6c-03ad-3458-4d04-3321d2205e39 2d6d9d6c-03ad-3458-4d04-3321d2205e39}" +2024-07-05T13:36:39.283Z [INFO] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: pipelining replication: peer="{Voter server-8308c6e0-dfce-43e3-3760-af609d174410 8308c6e0-dfce-43e3-3760-af609d174410}" +2024-07-05T13:36:40.287Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:39.282347843 +0000 UTC m=+65.278269294 (1 nodes), 669.504231ms from start of poll, 672.400719ms from cluster start. Timeout at 2024-07-05 13:36:40.287661899 +0000 UTC m=+66.283583351, 1.005314057s after stability +2024-07-05T13:36:40.289Z [INFO] creating new snapshot: path=/tmp/raft2948433730/snapshots/2-113-1720186600289.tmp +2024-07-05T13:36:40.291Z [INFO] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: copied to local snapshot: bytes=71 +2024-07-05T13:36:40.297Z [INFO] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: snapshot restore progress: id=2-113-1720186600289 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:40.298Z [INFO] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: restored user snapshot: index=113 +2024-07-05T13:36:40.299Z [ERROR] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: failed to get log: index=3 error="log not found" +2024-07-05T13:36:40.299Z [INFO] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: aborting pipeline replication: peer="{Voter server-2d6d9d6c-03ad-3458-4d04-3321d2205e39 2d6d9d6c-03ad-3458-4d04-3321d2205e39}" +2024-07-05T13:36:40.302Z [ERROR] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: failed to get log: index=3 error="log not found" +2024-07-05T13:36:40.302Z [INFO] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: aborting pipeline replication: peer="{Voter server-8308c6e0-dfce-43e3-3760-af609d174410 8308c6e0-dfce-43e3-3760-af609d174410}" +2024-07-05T13:36:40.308Z [ERROR] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: failed to get log: index=3 error="log not found" +2024-07-05T13:36:40.309Z [ERROR] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: failed to get log: index=3 error="log not found" +2024-07-05T13:36:40.311Z [INFO] creating new snapshot: path=/tmp/raft1279618532/snapshots/2-113-1720186600311.tmp +2024-07-05T13:36:40.312Z [INFO] creating new snapshot: path=/tmp/raft987353873/snapshots/2-113-1720186600312.tmp +2024-07-05T13:36:40.312Z [INFO] server-2d6d9d6c-03ad-3458-4d04-3321d2205e39: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:40.313Z [INFO] server-8308c6e0-dfce-43e3-3760-af609d174410: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:40.314Z [INFO] server-2d6d9d6c-03ad-3458-4d04-3321d2205e39: copied to local snapshot: bytes=71 +2024-07-05T13:36:40.314Z [INFO] server-8308c6e0-dfce-43e3-3760-af609d174410: copied to local snapshot: bytes=71 +2024-07-05T13:36:40.315Z [INFO] server-2d6d9d6c-03ad-3458-4d04-3321d2205e39: snapshot restore progress: id=2-113-1720186600311 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:40.315Z [INFO] server-2d6d9d6c-03ad-3458-4d04-3321d2205e39: Installed remote snapshot +2024-07-05T13:36:40.315Z [INFO] server-8308c6e0-dfce-43e3-3760-af609d174410: snapshot restore progress: id=2-113-1720186600312 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:40.316Z [INFO] server-8308c6e0-dfce-43e3-3760-af609d174410: Installed remote snapshot +2024-07-05T13:36:40.316Z [INFO] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: pipelining replication: peer="{Voter server-2d6d9d6c-03ad-3458-4d04-3321d2205e39 2d6d9d6c-03ad-3458-4d04-3321d2205e39}" +2024-07-05T13:36:40.317Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:36:40.317Z [INFO] server-0b43d00d-8fbd-a589-59ec-64dfb5c86ec0: pipelining replication: peer="{Voter server-8308c6e0-dfce-43e3-3760-af609d174410 8308c6e0-dfce-43e3-3760-af609d174410}" +2024-07-05T13:36:41.323Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:36:40.31720836 +0000 UTC m=+66.313129519 (2 nodes), 6.416µs from start of poll, 1.707260944s from cluster start. Timeout at 2024-07-05 13:36:41.322703538 +0000 UTC m=+67.318624989, 1.00549547s after stability +2024-07-05T13:36:41.341Z [INFO] server-a6986aa4-4362-0bd5-a187-93f91f2e2cd9: aborting pipeline replication: peer="{Voter server-89c91cb1-ee69-cb65-e0f5-65aa26ad15ff 89c91cb1-ee69-cb65-e0f5-65aa26ad15ff}" +2024-07-05T13:36:41.341Z [INFO] server-a6986aa4-4362-0bd5-a187-93f91f2e2cd9: aborting pipeline replication: peer="{Voter server-1be08a39-07e5-223c-70f3-5cf9f70d618d 1be08a39-07e5-223c-70f3-5cf9f70d618d}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true -2024-07-05T13:30:24.421Z [INFO] server-d898d79d-06c4-c51b-c750-ff057fb60ab4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d898d79d-06c4-c51b-c750-ff057fb60ab4 Address:d898d79d-06c4-c51b-c750-ff057fb60ab4} {Suffrage:Voter ID:server-f15517e0-6ebd-ac65-277c-9313f1039616 Address:f15517e0-6ebd-ac65-277c-9313f1039616} {Suffrage:Voter ID:server-67c5b37d-324f-f01f-5c67-7b0c42af48e8 Address:67c5b37d-324f-f01f-5c67-7b0c42af48e8}]" -2024-07-05T13:30:24.421Z [INFO] server-d898d79d-06c4-c51b-c750-ff057fb60ab4: entering follower state: follower="Node at d898d79d-06c4-c51b-c750-ff057fb60ab4 [Follower]" leader-address= leader-id= -2024-07-05T13:30:24.421Z [INFO] server-f15517e0-6ebd-ac65-277c-9313f1039616: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d898d79d-06c4-c51b-c750-ff057fb60ab4 Address:d898d79d-06c4-c51b-c750-ff057fb60ab4} {Suffrage:Voter ID:server-f15517e0-6ebd-ac65-277c-9313f1039616 Address:f15517e0-6ebd-ac65-277c-9313f1039616} {Suffrage:Voter ID:server-67c5b37d-324f-f01f-5c67-7b0c42af48e8 Address:67c5b37d-324f-f01f-5c67-7b0c42af48e8}]" -2024-07-05T13:30:24.422Z [INFO] server-f15517e0-6ebd-ac65-277c-9313f1039616: entering follower state: follower="Node at f15517e0-6ebd-ac65-277c-9313f1039616 [Follower]" leader-address= leader-id= -2024-07-05T13:30:24.422Z [INFO] server-67c5b37d-324f-f01f-5c67-7b0c42af48e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d898d79d-06c4-c51b-c750-ff057fb60ab4 Address:d898d79d-06c4-c51b-c750-ff057fb60ab4} {Suffrage:Voter ID:server-f15517e0-6ebd-ac65-277c-9313f1039616 Address:f15517e0-6ebd-ac65-277c-9313f1039616} {Suffrage:Voter ID:server-67c5b37d-324f-f01f-5c67-7b0c42af48e8 Address:67c5b37d-324f-f01f-5c67-7b0c42af48e8}]" -2024-07-05T13:30:24.422Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:24.422Z [INFO] server-67c5b37d-324f-f01f-5c67-7b0c42af48e8: entering follower state: follower="Node at 67c5b37d-324f-f01f-5c67-7b0c42af48e8 [Follower]" leader-address= leader-id= -2024-07-05T13:30:24.938Z [WARN] server-67c5b37d-324f-f01f-5c67-7b0c42af48e8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:24.938Z [INFO] server-67c5b37d-324f-f01f-5c67-7b0c42af48e8: entering candidate state: node="Node at 67c5b37d-324f-f01f-5c67-7b0c42af48e8 [Candidate]" term=2 -2024-07-05T13:30:24.938Z [INFO] server-67c5b37d-324f-f01f-5c67-7b0c42af48e8: election won: term=2 tally=2 -2024-07-05T13:30:24.938Z [INFO] server-67c5b37d-324f-f01f-5c67-7b0c42af48e8: entering leader state: leader="Node at 67c5b37d-324f-f01f-5c67-7b0c42af48e8 [Leader]" -2024-07-05T13:30:24.938Z [INFO] server-67c5b37d-324f-f01f-5c67-7b0c42af48e8: added peer, starting replication: peer=server-d898d79d-06c4-c51b-c750-ff057fb60ab4 -2024-07-05T13:30:24.938Z [INFO] server-67c5b37d-324f-f01f-5c67-7b0c42af48e8: added peer, starting replication: peer=server-f15517e0-6ebd-ac65-277c-9313f1039616 -2024-07-05T13:30:24.938Z [INFO] server-67c5b37d-324f-f01f-5c67-7b0c42af48e8: pipelining replication: peer="{Voter server-f15517e0-6ebd-ac65-277c-9313f1039616 f15517e0-6ebd-ac65-277c-9313f1039616}" -2024-07-05T13:30:24.939Z [INFO] server-67c5b37d-324f-f01f-5c67-7b0c42af48e8: pipelining replication: peer="{Voter server-d898d79d-06c4-c51b-c750-ff057fb60ab4 d898d79d-06c4-c51b-c750-ff057fb60ab4}" -2024-07-05T13:30:25.945Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:24.939516373 +0000 UTC m=+65.709427048 (1 nodes), 517.256493ms from start of poll, 518.227659ms from cluster start. Timeout at 2024-07-05 13:30:25.945318647 +0000 UTC m=+66.715229362, 1.005802314s after stability -2024-07-05T13:30:25.947Z [INFO] server-67c5b37d-324f-f01f-5c67-7b0c42af48e8: starting snapshot up to: index=12 -2024-07-05T13:30:25.947Z [INFO] creating new snapshot: path=/tmp/raft453820590/snapshots/2-12-1720186225947.tmp -2024-07-05T13:30:25.949Z [INFO] server-67c5b37d-324f-f01f-5c67-7b0c42af48e8: snapshot complete up to: index=12 -2024-07-05T13:30:25.953Z [INFO] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67 Address:1e9e43ca-d158-3d28-abfe-13c5b6e13f67} {Suffrage:Voter ID:server-04c4e0ac-f55e-28a6-2902-3f40311c9dee Address:04c4e0ac-f55e-28a6-2902-3f40311c9dee} {Suffrage:Voter ID:server-ceefe5af-8439-3684-0249-b0cf841966fa Address:ceefe5af-8439-3684-0249-b0cf841966fa}]" -2024-07-05T13:30:25.953Z [INFO] server-04c4e0ac-f55e-28a6-2902-3f40311c9dee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67 Address:1e9e43ca-d158-3d28-abfe-13c5b6e13f67} {Suffrage:Voter ID:server-04c4e0ac-f55e-28a6-2902-3f40311c9dee Address:04c4e0ac-f55e-28a6-2902-3f40311c9dee} {Suffrage:Voter ID:server-ceefe5af-8439-3684-0249-b0cf841966fa Address:ceefe5af-8439-3684-0249-b0cf841966fa}]" -2024-07-05T13:30:25.954Z [INFO] server-ceefe5af-8439-3684-0249-b0cf841966fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67 Address:1e9e43ca-d158-3d28-abfe-13c5b6e13f67} {Suffrage:Voter ID:server-04c4e0ac-f55e-28a6-2902-3f40311c9dee Address:04c4e0ac-f55e-28a6-2902-3f40311c9dee} {Suffrage:Voter ID:server-ceefe5af-8439-3684-0249-b0cf841966fa Address:ceefe5af-8439-3684-0249-b0cf841966fa}]" -2024-07-05T13:30:25.954Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:25.954Z [INFO] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: entering follower state: follower="Node at 1e9e43ca-d158-3d28-abfe-13c5b6e13f67 [Follower]" leader-address= leader-id= -2024-07-05T13:30:25.954Z [INFO] server-04c4e0ac-f55e-28a6-2902-3f40311c9dee: entering follower state: follower="Node at 04c4e0ac-f55e-28a6-2902-3f40311c9dee [Follower]" leader-address= leader-id= -2024-07-05T13:30:25.954Z [INFO] server-ceefe5af-8439-3684-0249-b0cf841966fa: entering follower state: follower="Node at ceefe5af-8439-3684-0249-b0cf841966fa [Follower]" leader-address= leader-id= -2024-07-05T13:30:26.529Z [WARN] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:26.529Z [INFO] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: entering candidate state: node="Node at 1e9e43ca-d158-3d28-abfe-13c5b6e13f67 [Candidate]" term=2 -2024-07-05T13:30:26.530Z [INFO] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: election won: term=2 tally=2 -2024-07-05T13:30:26.530Z [INFO] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: entering leader state: leader="Node at 1e9e43ca-d158-3d28-abfe-13c5b6e13f67 [Leader]" -2024-07-05T13:30:26.530Z [INFO] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: added peer, starting replication: peer=server-04c4e0ac-f55e-28a6-2902-3f40311c9dee -2024-07-05T13:30:26.530Z [INFO] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: added peer, starting replication: peer=server-ceefe5af-8439-3684-0249-b0cf841966fa -2024-07-05T13:30:26.530Z [INFO] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: pipelining replication: peer="{Voter server-ceefe5af-8439-3684-0249-b0cf841966fa ceefe5af-8439-3684-0249-b0cf841966fa}" -2024-07-05T13:30:26.531Z [INFO] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: pipelining replication: peer="{Voter server-04c4e0ac-f55e-28a6-2902-3f40311c9dee 04c4e0ac-f55e-28a6-2902-3f40311c9dee}" -2024-07-05T13:30:27.536Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:26.530854824 +0000 UTC m=+67.300765499 (1 nodes), 576.660129ms from start of poll, 577.399223ms from cluster start. Timeout at 2024-07-05 13:30:27.536686097 +0000 UTC m=+68.306596772, 1.005831273s after stability -2024-07-05T13:30:27.537Z [INFO] creating new snapshot: path=/tmp/raft2760585690/snapshots/2-113-1720186227537.tmp -2024-07-05T13:30:27.538Z [INFO] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: copied to local snapshot: bytes=71 -2024-07-05T13:30:27.538Z [INFO] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: snapshot restore progress: id=2-113-1720186227537 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:27.538Z [INFO] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: removing all old logs from log store -2024-07-05T13:30:27.538Z [INFO] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: compacting logs: from=1 to=2 -2024-07-05T13:30:27.538Z [INFO] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: restored user snapshot: index=113 -2024-07-05T13:30:27.539Z [ERROR] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: failed to get log: index=2 error="log not found" -2024-07-05T13:30:27.539Z [INFO] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: aborting pipeline replication: peer="{Voter server-ceefe5af-8439-3684-0249-b0cf841966fa ceefe5af-8439-3684-0249-b0cf841966fa}" -2024-07-05T13:30:27.539Z [ERROR] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: failed to get log: index=2 error="log not found" -2024-07-05T13:30:27.539Z [INFO] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: aborting pipeline replication: peer="{Voter server-04c4e0ac-f55e-28a6-2902-3f40311c9dee 04c4e0ac-f55e-28a6-2902-3f40311c9dee}" -2024-07-05T13:30:27.548Z [ERROR] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: failed to get log: index=2 error="log not found" -2024-07-05T13:30:27.549Z [INFO] creating new snapshot: path=/tmp/raft3031409443/snapshots/2-113-1720186227549.tmp -2024-07-05T13:30:27.549Z [INFO] server-ceefe5af-8439-3684-0249-b0cf841966fa: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:27.550Z [INFO] server-ceefe5af-8439-3684-0249-b0cf841966fa: copied to local snapshot: bytes=71 -2024-07-05T13:30:27.558Z [ERROR] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: failed to get log: index=2 error="log not found" -2024-07-05T13:30:27.559Z [INFO] creating new snapshot: path=/tmp/raft3484878559/snapshots/2-113-1720186227559.tmp -2024-07-05T13:30:27.559Z [INFO] server-04c4e0ac-f55e-28a6-2902-3f40311c9dee: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:27.560Z [INFO] server-04c4e0ac-f55e-28a6-2902-3f40311c9dee: copied to local snapshot: bytes=71 -2024-07-05T13:30:27.561Z [INFO] server-ceefe5af-8439-3684-0249-b0cf841966fa: snapshot restore progress: id=2-113-1720186227549 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:27.561Z [INFO] server-ceefe5af-8439-3684-0249-b0cf841966fa: removing all old logs from log store -2024-07-05T13:30:27.561Z [INFO] server-ceefe5af-8439-3684-0249-b0cf841966fa: compacting logs: from=1 to=2 -2024-07-05T13:30:27.561Z [INFO] server-ceefe5af-8439-3684-0249-b0cf841966fa: Installed remote snapshot -2024-07-05T13:30:27.561Z [INFO] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: pipelining replication: peer="{Voter server-ceefe5af-8439-3684-0249-b0cf841966fa ceefe5af-8439-3684-0249-b0cf841966fa}" -2024-07-05T13:30:27.561Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:27.564Z [INFO] server-04c4e0ac-f55e-28a6-2902-3f40311c9dee: snapshot restore progress: id=2-113-1720186227559 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:27.564Z [INFO] server-04c4e0ac-f55e-28a6-2902-3f40311c9dee: removing all old logs from log store -2024-07-05T13:30:27.564Z [INFO] server-04c4e0ac-f55e-28a6-2902-3f40311c9dee: compacting logs: from=1 to=2 -2024-07-05T13:30:27.564Z [INFO] server-04c4e0ac-f55e-28a6-2902-3f40311c9dee: Installed remote snapshot -2024-07-05T13:30:27.565Z [INFO] server-1e9e43ca-d158-3d28-abfe-13c5b6e13f67: pipelining replication: peer="{Voter server-04c4e0ac-f55e-28a6-2902-3f40311c9dee 04c4e0ac-f55e-28a6-2902-3f40311c9dee}" -2024-07-05T13:30:28.567Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:27.561599263 +0000 UTC m=+68.331509938 (2 nodes), 1.76µs from start of poll, 1.608143662s from cluster start. Timeout at 2024-07-05 13:30:28.567236022 +0000 UTC m=+69.337146657, 1.005636719s after stability -2024-07-05T13:30:28.581Z [INFO] server-67c5b37d-324f-f01f-5c67-7b0c42af48e8: aborting pipeline replication: peer="{Voter server-f15517e0-6ebd-ac65-277c-9313f1039616 f15517e0-6ebd-ac65-277c-9313f1039616}" -2024-07-05T13:30:28.581Z [INFO] server-67c5b37d-324f-f01f-5c67-7b0c42af48e8: aborting pipeline replication: peer="{Voter server-d898d79d-06c4-c51b-c750-ff057fb60ab4 d898d79d-06c4-c51b-c750-ff057fb60ab4}" +2024-07-05T13:36:41.353Z [INFO] server-07bb293e-2119-a233-88d9-682ff8bba341: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07bb293e-2119-a233-88d9-682ff8bba341 Address:07bb293e-2119-a233-88d9-682ff8bba341} {Suffrage:Voter ID:server-9c91f78b-8aee-1a0d-befa-6745a2bed300 Address:9c91f78b-8aee-1a0d-befa-6745a2bed300} {Suffrage:Voter ID:server-66065b68-40dd-1286-ffd0-0539a4bf660a Address:66065b68-40dd-1286-ffd0-0539a4bf660a}]" +2024-07-05T13:36:41.354Z [INFO] server-07bb293e-2119-a233-88d9-682ff8bba341: entering follower state: follower="Node at 07bb293e-2119-a233-88d9-682ff8bba341 [Follower]" leader-address= leader-id= +2024-07-05T13:36:41.354Z [INFO] server-9c91f78b-8aee-1a0d-befa-6745a2bed300: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07bb293e-2119-a233-88d9-682ff8bba341 Address:07bb293e-2119-a233-88d9-682ff8bba341} {Suffrage:Voter ID:server-9c91f78b-8aee-1a0d-befa-6745a2bed300 Address:9c91f78b-8aee-1a0d-befa-6745a2bed300} {Suffrage:Voter ID:server-66065b68-40dd-1286-ffd0-0539a4bf660a Address:66065b68-40dd-1286-ffd0-0539a4bf660a}]" +2024-07-05T13:36:41.354Z [INFO] server-9c91f78b-8aee-1a0d-befa-6745a2bed300: entering follower state: follower="Node at 9c91f78b-8aee-1a0d-befa-6745a2bed300 [Follower]" leader-address= leader-id= +2024-07-05T13:36:41.355Z [INFO] server-66065b68-40dd-1286-ffd0-0539a4bf660a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07bb293e-2119-a233-88d9-682ff8bba341 Address:07bb293e-2119-a233-88d9-682ff8bba341} {Suffrage:Voter ID:server-9c91f78b-8aee-1a0d-befa-6745a2bed300 Address:9c91f78b-8aee-1a0d-befa-6745a2bed300} {Suffrage:Voter ID:server-66065b68-40dd-1286-ffd0-0539a4bf660a Address:66065b68-40dd-1286-ffd0-0539a4bf660a}]" +2024-07-05T13:36:41.355Z [INFO] server-66065b68-40dd-1286-ffd0-0539a4bf660a: entering follower state: follower="Node at 66065b68-40dd-1286-ffd0-0539a4bf660a [Follower]" leader-address= leader-id= +2024-07-05T13:36:41.356Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:42.021Z [WARN] server-66065b68-40dd-1286-ffd0-0539a4bf660a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:42.022Z [INFO] server-66065b68-40dd-1286-ffd0-0539a4bf660a: entering candidate state: node="Node at 66065b68-40dd-1286-ffd0-0539a4bf660a [Candidate]" term=2 +2024-07-05T13:36:42.022Z [INFO] server-66065b68-40dd-1286-ffd0-0539a4bf660a: election won: term=2 tally=2 +2024-07-05T13:36:42.023Z [INFO] server-66065b68-40dd-1286-ffd0-0539a4bf660a: entering leader state: leader="Node at 66065b68-40dd-1286-ffd0-0539a4bf660a [Leader]" +2024-07-05T13:36:42.023Z [INFO] server-66065b68-40dd-1286-ffd0-0539a4bf660a: added peer, starting replication: peer=server-07bb293e-2119-a233-88d9-682ff8bba341 +2024-07-05T13:36:42.023Z [INFO] server-66065b68-40dd-1286-ffd0-0539a4bf660a: added peer, starting replication: peer=server-9c91f78b-8aee-1a0d-befa-6745a2bed300 +2024-07-05T13:36:42.024Z [INFO] server-66065b68-40dd-1286-ffd0-0539a4bf660a: pipelining replication: peer="{Voter server-07bb293e-2119-a233-88d9-682ff8bba341 07bb293e-2119-a233-88d9-682ff8bba341}" +2024-07-05T13:36:42.024Z [INFO] server-66065b68-40dd-1286-ffd0-0539a4bf660a: pipelining replication: peer="{Voter server-9c91f78b-8aee-1a0d-befa-6745a2bed300 9c91f78b-8aee-1a0d-befa-6745a2bed300}" +2024-07-05T13:36:43.028Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:42.023365366 +0000 UTC m=+68.019287401 (1 nodes), 667.178815ms from start of poll, 670.554212ms from cluster start. Timeout at 2024-07-05 13:36:43.028391262 +0000 UTC m=+69.024312130, 1.005024729s after stability +2024-07-05T13:36:43.031Z [INFO] server-66065b68-40dd-1286-ffd0-0539a4bf660a: starting snapshot up to: index=12 +2024-07-05T13:36:43.031Z [INFO] creating new snapshot: path=/tmp/raft2936307367/snapshots/2-12-1720186603031.tmp +2024-07-05T13:36:43.035Z [INFO] server-66065b68-40dd-1286-ffd0-0539a4bf660a: snapshot complete up to: index=12 +2024-07-05T13:36:43.046Z [INFO] server-aba3881d-d084-11bb-6129-0c7ea7474234: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aba3881d-d084-11bb-6129-0c7ea7474234 Address:aba3881d-d084-11bb-6129-0c7ea7474234} {Suffrage:Voter ID:server-685358ce-f601-049e-1846-90e8e156f79c Address:685358ce-f601-049e-1846-90e8e156f79c} {Suffrage:Voter ID:server-9d240966-84e9-4d62-3de5-9f207a8e84b3 Address:9d240966-84e9-4d62-3de5-9f207a8e84b3}]" +2024-07-05T13:36:43.046Z [INFO] server-aba3881d-d084-11bb-6129-0c7ea7474234: entering follower state: follower="Node at aba3881d-d084-11bb-6129-0c7ea7474234 [Follower]" leader-address= leader-id= +2024-07-05T13:36:43.047Z [INFO] server-685358ce-f601-049e-1846-90e8e156f79c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aba3881d-d084-11bb-6129-0c7ea7474234 Address:aba3881d-d084-11bb-6129-0c7ea7474234} {Suffrage:Voter ID:server-685358ce-f601-049e-1846-90e8e156f79c Address:685358ce-f601-049e-1846-90e8e156f79c} {Suffrage:Voter ID:server-9d240966-84e9-4d62-3de5-9f207a8e84b3 Address:9d240966-84e9-4d62-3de5-9f207a8e84b3}]" +2024-07-05T13:36:43.047Z [INFO] server-685358ce-f601-049e-1846-90e8e156f79c: entering follower state: follower="Node at 685358ce-f601-049e-1846-90e8e156f79c [Follower]" leader-address= leader-id= +2024-07-05T13:36:43.048Z [INFO] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aba3881d-d084-11bb-6129-0c7ea7474234 Address:aba3881d-d084-11bb-6129-0c7ea7474234} {Suffrage:Voter ID:server-685358ce-f601-049e-1846-90e8e156f79c Address:685358ce-f601-049e-1846-90e8e156f79c} {Suffrage:Voter ID:server-9d240966-84e9-4d62-3de5-9f207a8e84b3 Address:9d240966-84e9-4d62-3de5-9f207a8e84b3}]" +2024-07-05T13:36:43.048Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:43.048Z [INFO] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: entering follower state: follower="Node at 9d240966-84e9-4d62-3de5-9f207a8e84b3 [Follower]" leader-address= leader-id= +2024-07-05T13:36:43.586Z [WARN] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:43.586Z [INFO] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: entering candidate state: node="Node at 9d240966-84e9-4d62-3de5-9f207a8e84b3 [Candidate]" term=2 +2024-07-05T13:36:43.587Z [INFO] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: election won: term=2 tally=2 +2024-07-05T13:36:43.587Z [INFO] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: entering leader state: leader="Node at 9d240966-84e9-4d62-3de5-9f207a8e84b3 [Leader]" +2024-07-05T13:36:43.587Z [INFO] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: added peer, starting replication: peer=server-aba3881d-d084-11bb-6129-0c7ea7474234 +2024-07-05T13:36:43.587Z [INFO] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: added peer, starting replication: peer=server-685358ce-f601-049e-1846-90e8e156f79c +2024-07-05T13:36:43.588Z [INFO] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: pipelining replication: peer="{Voter server-685358ce-f601-049e-1846-90e8e156f79c 685358ce-f601-049e-1846-90e8e156f79c}" +2024-07-05T13:36:43.588Z [INFO] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: pipelining replication: peer="{Voter server-aba3881d-d084-11bb-6129-0c7ea7474234 aba3881d-d084-11bb-6129-0c7ea7474234}" +2024-07-05T13:36:44.592Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:43.587302179 +0000 UTC m=+69.583223048 (1 nodes), 539.043531ms from start of poll, 541.176159ms from cluster start. Timeout at 2024-07-05 13:36:44.592337408 +0000 UTC m=+70.588258859, 1.005035811s after stability +2024-07-05T13:36:44.594Z [INFO] creating new snapshot: path=/tmp/raft2809966352/snapshots/2-113-1720186604593.tmp +2024-07-05T13:36:44.598Z [INFO] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: copied to local snapshot: bytes=71 +2024-07-05T13:36:44.600Z [INFO] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: snapshot restore progress: id=2-113-1720186604593 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:44.600Z [INFO] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: removing all old logs from log store +2024-07-05T13:36:44.600Z [INFO] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: compacting logs: from=1 to=2 +2024-07-05T13:36:44.600Z [INFO] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: restored user snapshot: index=113 +2024-07-05T13:36:44.601Z [ERROR] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: failed to get log: index=2 error="log not found" +2024-07-05T13:36:44.601Z [INFO] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: aborting pipeline replication: peer="{Voter server-685358ce-f601-049e-1846-90e8e156f79c 685358ce-f601-049e-1846-90e8e156f79c}" +2024-07-05T13:36:44.601Z [ERROR] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: failed to get log: index=2 error="log not found" +2024-07-05T13:36:44.601Z [INFO] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: aborting pipeline replication: peer="{Voter server-aba3881d-d084-11bb-6129-0c7ea7474234 aba3881d-d084-11bb-6129-0c7ea7474234}" +2024-07-05T13:36:44.606Z [ERROR] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: failed to get log: index=2 error="log not found" +2024-07-05T13:36:44.608Z [ERROR] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: failed to get log: index=2 error="log not found" +2024-07-05T13:36:44.609Z [INFO] creating new snapshot: path=/tmp/raft859769845/snapshots/2-113-1720186604608.tmp +2024-07-05T13:36:44.625Z [INFO] server-685358ce-f601-049e-1846-90e8e156f79c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:44.627Z [INFO] creating new snapshot: path=/tmp/raft3724875115/snapshots/2-113-1720186604624.tmp +2024-07-05T13:36:44.628Z [INFO] server-685358ce-f601-049e-1846-90e8e156f79c: copied to local snapshot: bytes=71 +2024-07-05T13:36:44.629Z [INFO] server-aba3881d-d084-11bb-6129-0c7ea7474234: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:44.631Z [INFO] server-aba3881d-d084-11bb-6129-0c7ea7474234: copied to local snapshot: bytes=71 +2024-07-05T13:36:44.631Z [INFO] server-685358ce-f601-049e-1846-90e8e156f79c: snapshot restore progress: id=2-113-1720186604608 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:44.631Z [INFO] server-685358ce-f601-049e-1846-90e8e156f79c: removing all old logs from log store +2024-07-05T13:36:44.632Z [INFO] server-685358ce-f601-049e-1846-90e8e156f79c: compacting logs: from=1 to=2 +2024-07-05T13:36:44.632Z [INFO] server-685358ce-f601-049e-1846-90e8e156f79c: Installed remote snapshot +2024-07-05T13:36:44.633Z [INFO] server-aba3881d-d084-11bb-6129-0c7ea7474234: snapshot restore progress: id=2-113-1720186604624 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:44.633Z [INFO] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: pipelining replication: peer="{Voter server-685358ce-f601-049e-1846-90e8e156f79c 685358ce-f601-049e-1846-90e8e156f79c}" +2024-07-05T13:36:44.633Z [INFO] server-aba3881d-d084-11bb-6129-0c7ea7474234: removing all old logs from log store +2024-07-05T13:36:44.633Z [INFO] server-aba3881d-d084-11bb-6129-0c7ea7474234: compacting logs: from=1 to=2 +2024-07-05T13:36:44.633Z [INFO] server-aba3881d-d084-11bb-6129-0c7ea7474234: Installed remote snapshot +2024-07-05T13:36:44.634Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:36:44.635Z [INFO] server-9d240966-84e9-4d62-3de5-9f207a8e84b3: pipelining replication: peer="{Voter server-aba3881d-d084-11bb-6129-0c7ea7474234 aba3881d-d084-11bb-6129-0c7ea7474234}" +2024-07-05T13:36:45.648Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:36:44.634360849 +0000 UTC m=+70.630282300 (2 nodes), 7.583µs from start of poll, 1.588235411s from cluster start. Timeout at 2024-07-05 13:36:45.641197377 +0000 UTC m=+71.637118829, 1.006836529s after stability +2024-07-05T13:36:45.669Z [INFO] server-66065b68-40dd-1286-ffd0-0539a4bf660a: aborting pipeline replication: peer="{Voter server-9c91f78b-8aee-1a0d-befa-6745a2bed300 9c91f78b-8aee-1a0d-befa-6745a2bed300}" +2024-07-05T13:36:45.669Z [INFO] server-66065b68-40dd-1286-ffd0-0539a4bf660a: aborting pipeline replication: peer="{Voter server-07bb293e-2119-a233-88d9-682ff8bba341 07bb293e-2119-a233-88d9-682ff8bba341}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false -2024-07-05T13:30:28.583Z [INFO] server-e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a Address:e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a} {Suffrage:Voter ID:server-2eb81a4e-da3f-6737-4715-92c3d927acb7 Address:2eb81a4e-da3f-6737-4715-92c3d927acb7} {Suffrage:Voter ID:server-3c0899b8-322c-b2a5-53b7-3d3e103badff Address:3c0899b8-322c-b2a5-53b7-3d3e103badff}]" -2024-07-05T13:30:28.584Z [INFO] server-e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a: entering follower state: follower="Node at e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a [Follower]" leader-address= leader-id= -2024-07-05T13:30:28.584Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a Address:e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a} {Suffrage:Voter ID:server-2eb81a4e-da3f-6737-4715-92c3d927acb7 Address:2eb81a4e-da3f-6737-4715-92c3d927acb7} {Suffrage:Voter ID:server-3c0899b8-322c-b2a5-53b7-3d3e103badff Address:3c0899b8-322c-b2a5-53b7-3d3e103badff}]" -2024-07-05T13:30:28.584Z [INFO] server-3c0899b8-322c-b2a5-53b7-3d3e103badff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a Address:e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a} {Suffrage:Voter ID:server-2eb81a4e-da3f-6737-4715-92c3d927acb7 Address:2eb81a4e-da3f-6737-4715-92c3d927acb7} {Suffrage:Voter ID:server-3c0899b8-322c-b2a5-53b7-3d3e103badff Address:3c0899b8-322c-b2a5-53b7-3d3e103badff}]" -2024-07-05T13:30:28.585Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:28.585Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: entering follower state: follower="Node at 2eb81a4e-da3f-6737-4715-92c3d927acb7 [Follower]" leader-address= leader-id= -2024-07-05T13:30:28.585Z [INFO] server-3c0899b8-322c-b2a5-53b7-3d3e103badff: entering follower state: follower="Node at 3c0899b8-322c-b2a5-53b7-3d3e103badff [Follower]" leader-address= leader-id= -2024-07-05T13:30:29.101Z [WARN] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:29.101Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: entering candidate state: node="Node at 2eb81a4e-da3f-6737-4715-92c3d927acb7 [Candidate]" term=2 -2024-07-05T13:30:29.102Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: election won: term=2 tally=2 -2024-07-05T13:30:29.102Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: entering leader state: leader="Node at 2eb81a4e-da3f-6737-4715-92c3d927acb7 [Leader]" -2024-07-05T13:30:29.102Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: added peer, starting replication: peer=server-e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a -2024-07-05T13:30:29.102Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: added peer, starting replication: peer=server-3c0899b8-322c-b2a5-53b7-3d3e103badff -2024-07-05T13:30:29.102Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: pipelining replication: peer="{Voter server-3c0899b8-322c-b2a5-53b7-3d3e103badff 3c0899b8-322c-b2a5-53b7-3d3e103badff}" -2024-07-05T13:30:29.102Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: pipelining replication: peer="{Voter server-e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a}" -2024-07-05T13:30:30.111Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:29.102494563 +0000 UTC m=+69.872405238 (1 nodes), 517.468485ms from start of poll, 518.855197ms from cluster start. Timeout at 2024-07-05 13:30:30.108734461 +0000 UTC m=+70.878645136, 1.006239898s after stability -2024-07-05T13:30:30.112Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: starting snapshot up to: index=12 -2024-07-05T13:30:30.112Z [INFO] creating new snapshot: path=/tmp/raft980636946/snapshots/2-12-1720186230112.tmp -2024-07-05T13:30:30.113Z [WARN] server-3c0899b8-322c-b2a5-53b7-3d3e103badff: skipping application of old log: index=2 -2024-07-05T13:30:30.116Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: snapshot complete up to: index=12 -2024-07-05T13:30:30.119Z [INFO] creating new snapshot: path=/tmp/raft980636946/snapshots/2-1013-1720186230119.tmp -2024-07-05T13:30:30.121Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: copied to local snapshot: bytes=71 -2024-07-05T13:30:30.121Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: snapshot restore progress: id=2-1013-1720186230119 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:30.122Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: restored user snapshot: index=1013 -2024-07-05T13:30:30.122Z [ERROR] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: failed to get log: index=23 error="log not found" -2024-07-05T13:30:30.122Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: aborting pipeline replication: peer="{Voter server-3c0899b8-322c-b2a5-53b7-3d3e103badff 3c0899b8-322c-b2a5-53b7-3d3e103badff}" -2024-07-05T13:30:30.122Z [ERROR] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: failed to get log: index=23 error="log not found" -2024-07-05T13:30:30.122Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: aborting pipeline replication: peer="{Voter server-e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a}" -2024-07-05T13:30:30.129Z [ERROR] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: failed to get log: index=23 error="log not found" -2024-07-05T13:30:30.130Z [INFO] creating new snapshot: path=/tmp/raft804555380/snapshots/2-1013-1720186230130.tmp -2024-07-05T13:30:30.130Z [INFO] server-3c0899b8-322c-b2a5-53b7-3d3e103badff: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:30.131Z [INFO] server-3c0899b8-322c-b2a5-53b7-3d3e103badff: copied to local snapshot: bytes=71 -2024-07-05T13:30:30.132Z [ERROR] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: failed to get log: index=23 error="log not found" -2024-07-05T13:30:30.133Z [INFO] creating new snapshot: path=/tmp/raft928817388/snapshots/2-1013-1720186230133.tmp -2024-07-05T13:30:30.133Z [INFO] server-e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:30.134Z [INFO] server-e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a: copied to local snapshot: bytes=71 -2024-07-05T13:30:30.135Z [INFO] server-e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a: snapshot restore progress: id=2-1013-1720186230133 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:30.135Z [INFO] server-e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a: Installed remote snapshot -2024-07-05T13:30:30.135Z [WARN] server-e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a: skipping application of old log: index=22 -2024-07-05T13:30:30.135Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: pipelining replication: peer="{Voter server-e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a}" -2024-07-05T13:30:30.135Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:30.136Z [INFO] server-3c0899b8-322c-b2a5-53b7-3d3e103badff: snapshot restore progress: id=2-1013-1720186230130 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:30.136Z [INFO] server-3c0899b8-322c-b2a5-53b7-3d3e103badff: Installed remote snapshot -2024-07-05T13:30:30.136Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: pipelining replication: peer="{Voter server-3c0899b8-322c-b2a5-53b7-3d3e103badff 3c0899b8-322c-b2a5-53b7-3d3e103badff}" -2024-07-05T13:30:31.141Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:30.135634957 +0000 UTC m=+70.905545632 (2 nodes), 2.24µs from start of poll, 1.551995591s from cluster start. Timeout at 2024-07-05 13:30:31.141268116 +0000 UTC m=+71.911178791, 1.005633159s after stability -2024-07-05T13:30:31.155Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: aborting pipeline replication: peer="{Voter server-3c0899b8-322c-b2a5-53b7-3d3e103badff 3c0899b8-322c-b2a5-53b7-3d3e103badff}" -2024-07-05T13:30:31.155Z [INFO] server-2eb81a4e-da3f-6737-4715-92c3d927acb7: aborting pipeline replication: peer="{Voter server-e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a e1bb0dc0-ab7c-f8e8-dadd-b87434c1603a}" +2024-07-05T13:36:45.682Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dae977b2-6021-a730-f1c0-4d298b791540 Address:dae977b2-6021-a730-f1c0-4d298b791540} {Suffrage:Voter ID:server-f76259f0-7f8e-f355-0ce8-0fd96fd46b59 Address:f76259f0-7f8e-f355-0ce8-0fd96fd46b59} {Suffrage:Voter ID:server-2f9fa80f-0807-1aaa-7d93-084e6b46d4fd Address:2f9fa80f-0807-1aaa-7d93-084e6b46d4fd}]" +2024-07-05T13:36:45.682Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: entering follower state: follower="Node at dae977b2-6021-a730-f1c0-4d298b791540 [Follower]" leader-address= leader-id= +2024-07-05T13:36:45.683Z [INFO] server-f76259f0-7f8e-f355-0ce8-0fd96fd46b59: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dae977b2-6021-a730-f1c0-4d298b791540 Address:dae977b2-6021-a730-f1c0-4d298b791540} {Suffrage:Voter ID:server-f76259f0-7f8e-f355-0ce8-0fd96fd46b59 Address:f76259f0-7f8e-f355-0ce8-0fd96fd46b59} {Suffrage:Voter ID:server-2f9fa80f-0807-1aaa-7d93-084e6b46d4fd Address:2f9fa80f-0807-1aaa-7d93-084e6b46d4fd}]" +2024-07-05T13:36:45.683Z [INFO] server-f76259f0-7f8e-f355-0ce8-0fd96fd46b59: entering follower state: follower="Node at f76259f0-7f8e-f355-0ce8-0fd96fd46b59 [Follower]" leader-address= leader-id= +2024-07-05T13:36:45.683Z [INFO] server-2f9fa80f-0807-1aaa-7d93-084e6b46d4fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dae977b2-6021-a730-f1c0-4d298b791540 Address:dae977b2-6021-a730-f1c0-4d298b791540} {Suffrage:Voter ID:server-f76259f0-7f8e-f355-0ce8-0fd96fd46b59 Address:f76259f0-7f8e-f355-0ce8-0fd96fd46b59} {Suffrage:Voter ID:server-2f9fa80f-0807-1aaa-7d93-084e6b46d4fd Address:2f9fa80f-0807-1aaa-7d93-084e6b46d4fd}]" +2024-07-05T13:36:45.684Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:45.684Z [INFO] server-2f9fa80f-0807-1aaa-7d93-084e6b46d4fd: entering follower state: follower="Node at 2f9fa80f-0807-1aaa-7d93-084e6b46d4fd [Follower]" leader-address= leader-id= +2024-07-05T13:36:46.326Z [WARN] server-dae977b2-6021-a730-f1c0-4d298b791540: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:46.326Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: entering candidate state: node="Node at dae977b2-6021-a730-f1c0-4d298b791540 [Candidate]" term=2 +2024-07-05T13:36:46.327Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: election won: term=2 tally=2 +2024-07-05T13:36:46.327Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: entering leader state: leader="Node at dae977b2-6021-a730-f1c0-4d298b791540 [Leader]" +2024-07-05T13:36:46.327Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: added peer, starting replication: peer=server-f76259f0-7f8e-f355-0ce8-0fd96fd46b59 +2024-07-05T13:36:46.327Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: added peer, starting replication: peer=server-2f9fa80f-0807-1aaa-7d93-084e6b46d4fd +2024-07-05T13:36:46.328Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: pipelining replication: peer="{Voter server-f76259f0-7f8e-f355-0ce8-0fd96fd46b59 f76259f0-7f8e-f355-0ce8-0fd96fd46b59}" +2024-07-05T13:36:46.329Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: pipelining replication: peer="{Voter server-2f9fa80f-0807-1aaa-7d93-084e6b46d4fd 2f9fa80f-0807-1aaa-7d93-084e6b46d4fd}" +2024-07-05T13:36:47.333Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:46.328396119 +0000 UTC m=+72.324317571 (1 nodes), 644.261234ms from start of poll, 646.570026ms from cluster start. Timeout at 2024-07-05 13:36:47.333422889 +0000 UTC m=+73.329344341, 1.00502677s after stability +2024-07-05T13:36:47.339Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: starting snapshot up to: index=12 +2024-07-05T13:36:47.339Z [INFO] creating new snapshot: path=/tmp/raft1082672546/snapshots/2-12-1720186607339.tmp +2024-07-05T13:36:47.349Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: snapshot complete up to: index=12 +2024-07-05T13:36:47.355Z [INFO] creating new snapshot: path=/tmp/raft1082672546/snapshots/2-1013-1720186607355.tmp +2024-07-05T13:36:47.360Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: copied to local snapshot: bytes=71 +2024-07-05T13:36:47.361Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: snapshot restore progress: id=2-1013-1720186607355 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:47.361Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: restored user snapshot: index=1013 +2024-07-05T13:36:47.362Z [ERROR] server-dae977b2-6021-a730-f1c0-4d298b791540: failed to get log: index=23 error="log not found" +2024-07-05T13:36:47.362Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: aborting pipeline replication: peer="{Voter server-2f9fa80f-0807-1aaa-7d93-084e6b46d4fd 2f9fa80f-0807-1aaa-7d93-084e6b46d4fd}" +2024-07-05T13:36:47.362Z [ERROR] server-dae977b2-6021-a730-f1c0-4d298b791540: failed to get log: index=23 error="log not found" +2024-07-05T13:36:47.362Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: aborting pipeline replication: peer="{Voter server-f76259f0-7f8e-f355-0ce8-0fd96fd46b59 f76259f0-7f8e-f355-0ce8-0fd96fd46b59}" +2024-07-05T13:36:47.368Z [ERROR] server-dae977b2-6021-a730-f1c0-4d298b791540: failed to get log: index=23 error="log not found" +2024-07-05T13:36:47.369Z [ERROR] server-dae977b2-6021-a730-f1c0-4d298b791540: failed to get log: index=23 error="log not found" +2024-07-05T13:36:47.370Z [INFO] creating new snapshot: path=/tmp/raft2115810838/snapshots/2-1013-1720186607370.tmp +2024-07-05T13:36:47.371Z [INFO] creating new snapshot: path=/tmp/raft2569226033/snapshots/2-1013-1720186607371.tmp +2024-07-05T13:36:47.372Z [INFO] server-2f9fa80f-0807-1aaa-7d93-084e6b46d4fd: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:47.372Z [INFO] server-f76259f0-7f8e-f355-0ce8-0fd96fd46b59: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:47.373Z [INFO] server-2f9fa80f-0807-1aaa-7d93-084e6b46d4fd: copied to local snapshot: bytes=71 +2024-07-05T13:36:47.374Z [INFO] server-2f9fa80f-0807-1aaa-7d93-084e6b46d4fd: snapshot restore progress: id=2-1013-1720186607370 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:47.374Z [INFO] server-2f9fa80f-0807-1aaa-7d93-084e6b46d4fd: Installed remote snapshot +2024-07-05T13:36:47.375Z [INFO] server-f76259f0-7f8e-f355-0ce8-0fd96fd46b59: copied to local snapshot: bytes=71 +2024-07-05T13:36:47.375Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: pipelining replication: peer="{Voter server-2f9fa80f-0807-1aaa-7d93-084e6b46d4fd 2f9fa80f-0807-1aaa-7d93-084e6b46d4fd}" +2024-07-05T13:36:47.375Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:36:47.375Z [INFO] server-f76259f0-7f8e-f355-0ce8-0fd96fd46b59: snapshot restore progress: id=2-1013-1720186607371 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:47.375Z [INFO] server-f76259f0-7f8e-f355-0ce8-0fd96fd46b59: Installed remote snapshot +2024-07-05T13:36:47.376Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: pipelining replication: peer="{Voter server-f76259f0-7f8e-f355-0ce8-0fd96fd46b59 f76259f0-7f8e-f355-0ce8-0fd96fd46b59}" +2024-07-05T13:36:48.380Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:36:47.375719325 +0000 UTC m=+73.371639902 (2 nodes), 4.375µs from start of poll, 1.693892357s from cluster start. Timeout at 2024-07-05 13:36:48.380749012 +0000 UTC m=+74.376670172, 1.00503027s after stability +2024-07-05T13:36:48.398Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: aborting pipeline replication: peer="{Voter server-2f9fa80f-0807-1aaa-7d93-084e6b46d4fd 2f9fa80f-0807-1aaa-7d93-084e6b46d4fd}" +2024-07-05T13:36:48.398Z [INFO] server-dae977b2-6021-a730-f1c0-4d298b791540: aborting pipeline replication: peer="{Voter server-f76259f0-7f8e-f355-0ce8-0fd96fd46b59 f76259f0-7f8e-f355-0ce8-0fd96fd46b59}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false -2024-07-05T13:30:31.158Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df Address:2c139ea1-ae9c-b9c2-ea6a-9d89237367df} {Suffrage:Voter ID:server-8bd3adeb-daee-39af-ec05-dfcd334b460f Address:8bd3adeb-daee-39af-ec05-dfcd334b460f} {Suffrage:Voter ID:server-ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8 Address:ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8}]" -2024-07-05T13:30:31.158Z [INFO] server-8bd3adeb-daee-39af-ec05-dfcd334b460f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df Address:2c139ea1-ae9c-b9c2-ea6a-9d89237367df} {Suffrage:Voter ID:server-8bd3adeb-daee-39af-ec05-dfcd334b460f Address:8bd3adeb-daee-39af-ec05-dfcd334b460f} {Suffrage:Voter ID:server-ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8 Address:ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8}]" -2024-07-05T13:30:31.159Z [INFO] server-ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df Address:2c139ea1-ae9c-b9c2-ea6a-9d89237367df} {Suffrage:Voter ID:server-8bd3adeb-daee-39af-ec05-dfcd334b460f Address:8bd3adeb-daee-39af-ec05-dfcd334b460f} {Suffrage:Voter ID:server-ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8 Address:ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8}]" -2024-07-05T13:30:31.159Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:31.159Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: entering follower state: follower="Node at 2c139ea1-ae9c-b9c2-ea6a-9d89237367df [Follower]" leader-address= leader-id= -2024-07-05T13:30:31.159Z [INFO] server-8bd3adeb-daee-39af-ec05-dfcd334b460f: entering follower state: follower="Node at 8bd3adeb-daee-39af-ec05-dfcd334b460f [Follower]" leader-address= leader-id= -2024-07-05T13:30:31.159Z [INFO] server-ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8: entering follower state: follower="Node at ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8 [Follower]" leader-address= leader-id= -2024-07-05T13:30:31.670Z [WARN] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:31.670Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: entering candidate state: node="Node at 2c139ea1-ae9c-b9c2-ea6a-9d89237367df [Candidate]" term=2 -2024-07-05T13:30:31.670Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: election won: term=2 tally=2 -2024-07-05T13:30:31.670Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: entering leader state: leader="Node at 2c139ea1-ae9c-b9c2-ea6a-9d89237367df [Leader]" -2024-07-05T13:30:31.670Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: added peer, starting replication: peer=server-8bd3adeb-daee-39af-ec05-dfcd334b460f -2024-07-05T13:30:31.670Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: added peer, starting replication: peer=server-ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8 -2024-07-05T13:30:31.670Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: pipelining replication: peer="{Voter server-ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8 ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8}" -2024-07-05T13:30:31.670Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: pipelining replication: peer="{Voter server-8bd3adeb-daee-39af-ec05-dfcd334b460f 8bd3adeb-daee-39af-ec05-dfcd334b460f}" -2024-07-05T13:30:32.688Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:31.671133247 +0000 UTC m=+72.441043922 (1 nodes), 512.002797ms from start of poll, 512.634135ms from cluster start. Timeout at 2024-07-05 13:30:32.676912481 +0000 UTC m=+73.446823156, 1.005779234s after stability -2024-07-05T13:30:32.690Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: starting snapshot up to: index=12 -2024-07-05T13:30:32.690Z [INFO] creating new snapshot: path=/tmp/raft30487931/snapshots/2-12-1720186232690.tmp -2024-07-05T13:30:32.691Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: snapshot complete up to: index=12 -2024-07-05T13:30:32.692Z [WARN] server-ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8: skipping application of old log: index=2 -2024-07-05T13:30:32.693Z [INFO] creating new snapshot: path=/tmp/raft30487931/snapshots/2-1013-1720186232693.tmp -2024-07-05T13:30:32.695Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: copied to local snapshot: bytes=71 -2024-07-05T13:30:32.695Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: snapshot restore progress: id=2-1013-1720186232693 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:32.695Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: removing all old logs from log store -2024-07-05T13:30:32.695Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: compacting logs: from=1 to=22 -2024-07-05T13:30:32.695Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: restored user snapshot: index=1013 -2024-07-05T13:30:32.696Z [ERROR] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: failed to get log: index=22 error="log not found" -2024-07-05T13:30:32.696Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: aborting pipeline replication: peer="{Voter server-8bd3adeb-daee-39af-ec05-dfcd334b460f 8bd3adeb-daee-39af-ec05-dfcd334b460f}" -2024-07-05T13:30:32.696Z [ERROR] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: failed to get log: index=22 error="log not found" -2024-07-05T13:30:32.696Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: aborting pipeline replication: peer="{Voter server-ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8 ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8}" -2024-07-05T13:30:32.703Z [ERROR] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: failed to get log: index=22 error="log not found" -2024-07-05T13:30:32.704Z [INFO] creating new snapshot: path=/tmp/raft404064143/snapshots/2-1013-1720186232704.tmp -2024-07-05T13:30:32.705Z [INFO] server-8bd3adeb-daee-39af-ec05-dfcd334b460f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:32.706Z [INFO] server-8bd3adeb-daee-39af-ec05-dfcd334b460f: copied to local snapshot: bytes=71 -2024-07-05T13:30:32.706Z [INFO] server-8bd3adeb-daee-39af-ec05-dfcd334b460f: snapshot restore progress: id=2-1013-1720186232704 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:32.706Z [INFO] server-8bd3adeb-daee-39af-ec05-dfcd334b460f: removing all old logs from log store -2024-07-05T13:30:32.706Z [INFO] server-8bd3adeb-daee-39af-ec05-dfcd334b460f: compacting logs: from=1 to=22 -2024-07-05T13:30:32.706Z [INFO] server-8bd3adeb-daee-39af-ec05-dfcd334b460f: Installed remote snapshot -2024-07-05T13:30:32.706Z [ERROR] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: failed to get log: index=22 error="log not found" -2024-07-05T13:30:32.707Z [WARN] server-8bd3adeb-daee-39af-ec05-dfcd334b460f: skipping application of old log: index=22 -2024-07-05T13:30:32.707Z [INFO] creating new snapshot: path=/tmp/raft3020103747/snapshots/2-1013-1720186232707.tmp -2024-07-05T13:30:32.708Z [INFO] server-ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:32.709Z [INFO] server-ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8: copied to local snapshot: bytes=71 -2024-07-05T13:30:32.710Z [INFO] server-ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8: snapshot restore progress: id=2-1013-1720186232707 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:32.710Z [INFO] server-ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8: removing all old logs from log store -2024-07-05T13:30:32.710Z [INFO] server-ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8: compacting logs: from=1 to=22 -2024-07-05T13:30:32.710Z [INFO] server-ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8: Installed remote snapshot -2024-07-05T13:30:32.710Z [WARN] server-ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8: skipping application of old log: index=22 -2024-07-05T13:30:32.710Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: pipelining replication: peer="{Voter server-ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8 ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8}" -2024-07-05T13:30:32.710Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:32.711Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: pipelining replication: peer="{Voter server-8bd3adeb-daee-39af-ec05-dfcd334b460f 8bd3adeb-daee-39af-ec05-dfcd334b460f}" -2024-07-05T13:30:33.716Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:32.710760133 +0000 UTC m=+73.480670808 (2 nodes), 1.52µs from start of poll, 1.552261021s from cluster start. Timeout at 2024-07-05 13:30:33.716770319 +0000 UTC m=+74.486680994, 1.006010186s after stability -2024-07-05T13:30:33.729Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: aborting pipeline replication: peer="{Voter server-ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8 ba5249d8-fe44-ab88-6e5e-52ad5f7b27b8}" -2024-07-05T13:30:33.729Z [INFO] server-2c139ea1-ae9c-b9c2-ea6a-9d89237367df: aborting pipeline replication: peer="{Voter server-8bd3adeb-daee-39af-ec05-dfcd334b460f 8bd3adeb-daee-39af-ec05-dfcd334b460f}" +2024-07-05T13:36:48.415Z [INFO] server-c948d69f-cf04-b800-3e56-d6ba23f2e2e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c948d69f-cf04-b800-3e56-d6ba23f2e2e4 Address:c948d69f-cf04-b800-3e56-d6ba23f2e2e4} {Suffrage:Voter ID:server-53552a3d-a624-826b-6c4a-ef11407c77d2 Address:53552a3d-a624-826b-6c4a-ef11407c77d2} {Suffrage:Voter ID:server-a4319bea-b102-dfa4-5181-9a3603dd8941 Address:a4319bea-b102-dfa4-5181-9a3603dd8941}]" +2024-07-05T13:36:48.416Z [INFO] server-c948d69f-cf04-b800-3e56-d6ba23f2e2e4: entering follower state: follower="Node at c948d69f-cf04-b800-3e56-d6ba23f2e2e4 [Follower]" leader-address= leader-id= +2024-07-05T13:36:48.416Z [INFO] server-53552a3d-a624-826b-6c4a-ef11407c77d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c948d69f-cf04-b800-3e56-d6ba23f2e2e4 Address:c948d69f-cf04-b800-3e56-d6ba23f2e2e4} {Suffrage:Voter ID:server-53552a3d-a624-826b-6c4a-ef11407c77d2 Address:53552a3d-a624-826b-6c4a-ef11407c77d2} {Suffrage:Voter ID:server-a4319bea-b102-dfa4-5181-9a3603dd8941 Address:a4319bea-b102-dfa4-5181-9a3603dd8941}]" +2024-07-05T13:36:48.416Z [INFO] server-53552a3d-a624-826b-6c4a-ef11407c77d2: entering follower state: follower="Node at 53552a3d-a624-826b-6c4a-ef11407c77d2 [Follower]" leader-address= leader-id= +2024-07-05T13:36:48.417Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c948d69f-cf04-b800-3e56-d6ba23f2e2e4 Address:c948d69f-cf04-b800-3e56-d6ba23f2e2e4} {Suffrage:Voter ID:server-53552a3d-a624-826b-6c4a-ef11407c77d2 Address:53552a3d-a624-826b-6c4a-ef11407c77d2} {Suffrage:Voter ID:server-a4319bea-b102-dfa4-5181-9a3603dd8941 Address:a4319bea-b102-dfa4-5181-9a3603dd8941}]" +2024-07-05T13:36:48.417Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:48.418Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: entering follower state: follower="Node at a4319bea-b102-dfa4-5181-9a3603dd8941 [Follower]" leader-address= leader-id= +2024-07-05T13:36:48.993Z [WARN] server-a4319bea-b102-dfa4-5181-9a3603dd8941: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:48.993Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: entering candidate state: node="Node at a4319bea-b102-dfa4-5181-9a3603dd8941 [Candidate]" term=2 +2024-07-05T13:36:48.993Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: election won: term=2 tally=2 +2024-07-05T13:36:48.994Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: entering leader state: leader="Node at a4319bea-b102-dfa4-5181-9a3603dd8941 [Leader]" +2024-07-05T13:36:48.994Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: added peer, starting replication: peer=server-c948d69f-cf04-b800-3e56-d6ba23f2e2e4 +2024-07-05T13:36:48.994Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: added peer, starting replication: peer=server-53552a3d-a624-826b-6c4a-ef11407c77d2 +2024-07-05T13:36:48.995Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: pipelining replication: peer="{Voter server-53552a3d-a624-826b-6c4a-ef11407c77d2 53552a3d-a624-826b-6c4a-ef11407c77d2}" +2024-07-05T13:36:48.995Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: pipelining replication: peer="{Voter server-c948d69f-cf04-b800-3e56-d6ba23f2e2e4 c948d69f-cf04-b800-3e56-d6ba23f2e2e4}" +2024-07-05T13:36:50.000Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:48.995189041 +0000 UTC m=+74.991110493 (1 nodes), 577.080545ms from start of poll, 580.394109ms from cluster start. Timeout at 2024-07-05 13:36:50.000324893 +0000 UTC m=+75.996246345, 1.005135852s after stability +2024-07-05T13:36:50.005Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: starting snapshot up to: index=12 +2024-07-05T13:36:50.005Z [INFO] creating new snapshot: path=/tmp/raft2158605251/snapshots/2-12-1720186610005.tmp +2024-07-05T13:36:50.008Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: snapshot complete up to: index=12 +2024-07-05T13:36:50.013Z [INFO] creating new snapshot: path=/tmp/raft2158605251/snapshots/2-1013-1720186610013.tmp +2024-07-05T13:36:50.017Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: copied to local snapshot: bytes=71 +2024-07-05T13:36:50.019Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: snapshot restore progress: id=2-1013-1720186610013 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:50.019Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: removing all old logs from log store +2024-07-05T13:36:50.019Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: compacting logs: from=1 to=22 +2024-07-05T13:36:50.019Z [ERROR] server-a4319bea-b102-dfa4-5181-9a3603dd8941: failed to get log: index=22 error="log not found" +2024-07-05T13:36:50.019Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: aborting pipeline replication: peer="{Voter server-53552a3d-a624-826b-6c4a-ef11407c77d2 53552a3d-a624-826b-6c4a-ef11407c77d2}" +2024-07-05T13:36:50.019Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: restored user snapshot: index=1013 +2024-07-05T13:36:50.020Z [ERROR] server-a4319bea-b102-dfa4-5181-9a3603dd8941: failed to get log: index=22 error="log not found" +2024-07-05T13:36:50.020Z [ERROR] server-a4319bea-b102-dfa4-5181-9a3603dd8941: failed to get log: index=22 error="log not found" +2024-07-05T13:36:50.020Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: aborting pipeline replication: peer="{Voter server-c948d69f-cf04-b800-3e56-d6ba23f2e2e4 c948d69f-cf04-b800-3e56-d6ba23f2e2e4}" +2024-07-05T13:36:50.022Z [INFO] creating new snapshot: path=/tmp/raft3180352079/snapshots/2-1013-1720186610022.tmp +2024-07-05T13:36:50.024Z [INFO] server-53552a3d-a624-826b-6c4a-ef11407c77d2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:50.025Z [INFO] server-53552a3d-a624-826b-6c4a-ef11407c77d2: copied to local snapshot: bytes=71 +2024-07-05T13:36:50.026Z [INFO] server-53552a3d-a624-826b-6c4a-ef11407c77d2: snapshot restore progress: id=2-1013-1720186610022 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:50.027Z [INFO] server-53552a3d-a624-826b-6c4a-ef11407c77d2: removing all old logs from log store +2024-07-05T13:36:50.027Z [INFO] server-53552a3d-a624-826b-6c4a-ef11407c77d2: compacting logs: from=1 to=22 +2024-07-05T13:36:50.027Z [INFO] server-53552a3d-a624-826b-6c4a-ef11407c77d2: Installed remote snapshot +2024-07-05T13:36:50.027Z [ERROR] server-a4319bea-b102-dfa4-5181-9a3603dd8941: failed to get log: index=22 error="log not found" +2024-07-05T13:36:50.027Z [WARN] server-53552a3d-a624-826b-6c4a-ef11407c77d2: skipping application of old log: index=22 +2024-07-05T13:36:50.027Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: pipelining replication: peer="{Voter server-53552a3d-a624-826b-6c4a-ef11407c77d2 53552a3d-a624-826b-6c4a-ef11407c77d2}" +2024-07-05T13:36:50.028Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:36:50.029Z [INFO] creating new snapshot: path=/tmp/raft3793594613/snapshots/2-1013-1720186610029.tmp +2024-07-05T13:36:50.030Z [INFO] server-c948d69f-cf04-b800-3e56-d6ba23f2e2e4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:50.032Z [INFO] server-c948d69f-cf04-b800-3e56-d6ba23f2e2e4: copied to local snapshot: bytes=71 +2024-07-05T13:36:50.034Z [INFO] server-c948d69f-cf04-b800-3e56-d6ba23f2e2e4: snapshot restore progress: id=2-1013-1720186610029 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:50.034Z [INFO] server-c948d69f-cf04-b800-3e56-d6ba23f2e2e4: removing all old logs from log store +2024-07-05T13:36:50.034Z [INFO] server-c948d69f-cf04-b800-3e56-d6ba23f2e2e4: compacting logs: from=1 to=22 +2024-07-05T13:36:50.034Z [INFO] server-c948d69f-cf04-b800-3e56-d6ba23f2e2e4: Installed remote snapshot +2024-07-05T13:36:50.035Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: pipelining replication: peer="{Voter server-c948d69f-cf04-b800-3e56-d6ba23f2e2e4 c948d69f-cf04-b800-3e56-d6ba23f2e2e4}" +2024-07-05T13:36:51.033Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:36:50.028370173 +0000 UTC m=+76.024291333 (2 nodes), 5.542µs from start of poll, 1.613574949s from cluster start. Timeout at 2024-07-05 13:36:51.033397527 +0000 UTC m=+77.029318687, 1.005027354s after stability +2024-07-05T13:36:51.048Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: aborting pipeline replication: peer="{Voter server-53552a3d-a624-826b-6c4a-ef11407c77d2 53552a3d-a624-826b-6c4a-ef11407c77d2}" +2024-07-05T13:36:51.048Z [INFO] server-a4319bea-b102-dfa4-5181-9a3603dd8941: aborting pipeline replication: peer="{Voter server-c948d69f-cf04-b800-3e56-d6ba23f2e2e4 c948d69f-cf04-b800-3e56-d6ba23f2e2e4}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true -2024-07-05T13:30:33.734Z [INFO] server-962bbec8-bdd6-c589-a7bb-bcddc12c8a72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-962bbec8-bdd6-c589-a7bb-bcddc12c8a72 Address:962bbec8-bdd6-c589-a7bb-bcddc12c8a72} {Suffrage:Voter ID:server-4daca39e-d423-537b-0066-2cfae97031d4 Address:4daca39e-d423-537b-0066-2cfae97031d4} {Suffrage:Voter ID:server-a37db21b-fa37-cb02-7d08-0e0fb48e698b Address:a37db21b-fa37-cb02-7d08-0e0fb48e698b}]" -2024-07-05T13:30:33.734Z [INFO] server-4daca39e-d423-537b-0066-2cfae97031d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-962bbec8-bdd6-c589-a7bb-bcddc12c8a72 Address:962bbec8-bdd6-c589-a7bb-bcddc12c8a72} {Suffrage:Voter ID:server-4daca39e-d423-537b-0066-2cfae97031d4 Address:4daca39e-d423-537b-0066-2cfae97031d4} {Suffrage:Voter ID:server-a37db21b-fa37-cb02-7d08-0e0fb48e698b Address:a37db21b-fa37-cb02-7d08-0e0fb48e698b}]" -2024-07-05T13:30:33.734Z [INFO] server-a37db21b-fa37-cb02-7d08-0e0fb48e698b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-962bbec8-bdd6-c589-a7bb-bcddc12c8a72 Address:962bbec8-bdd6-c589-a7bb-bcddc12c8a72} {Suffrage:Voter ID:server-4daca39e-d423-537b-0066-2cfae97031d4 Address:4daca39e-d423-537b-0066-2cfae97031d4} {Suffrage:Voter ID:server-a37db21b-fa37-cb02-7d08-0e0fb48e698b Address:a37db21b-fa37-cb02-7d08-0e0fb48e698b}]" -2024-07-05T13:30:33.734Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:33.734Z [INFO] server-962bbec8-bdd6-c589-a7bb-bcddc12c8a72: entering follower state: follower="Node at 962bbec8-bdd6-c589-a7bb-bcddc12c8a72 [Follower]" leader-address= leader-id= -2024-07-05T13:30:33.734Z [INFO] server-4daca39e-d423-537b-0066-2cfae97031d4: entering follower state: follower="Node at 4daca39e-d423-537b-0066-2cfae97031d4 [Follower]" leader-address= leader-id= -2024-07-05T13:30:33.734Z [INFO] server-a37db21b-fa37-cb02-7d08-0e0fb48e698b: entering follower state: follower="Node at a37db21b-fa37-cb02-7d08-0e0fb48e698b [Follower]" leader-address= leader-id= -2024-07-05T13:30:34.370Z [WARN] server-4daca39e-d423-537b-0066-2cfae97031d4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:34.370Z [INFO] server-4daca39e-d423-537b-0066-2cfae97031d4: entering candidate state: node="Node at 4daca39e-d423-537b-0066-2cfae97031d4 [Candidate]" term=2 -2024-07-05T13:30:34.370Z [INFO] server-4daca39e-d423-537b-0066-2cfae97031d4: election won: term=2 tally=2 -2024-07-05T13:30:34.370Z [INFO] server-4daca39e-d423-537b-0066-2cfae97031d4: entering leader state: leader="Node at 4daca39e-d423-537b-0066-2cfae97031d4 [Leader]" -2024-07-05T13:30:34.371Z [INFO] server-4daca39e-d423-537b-0066-2cfae97031d4: added peer, starting replication: peer=server-962bbec8-bdd6-c589-a7bb-bcddc12c8a72 -2024-07-05T13:30:34.371Z [INFO] server-4daca39e-d423-537b-0066-2cfae97031d4: added peer, starting replication: peer=server-a37db21b-fa37-cb02-7d08-0e0fb48e698b -2024-07-05T13:30:34.371Z [INFO] server-4daca39e-d423-537b-0066-2cfae97031d4: pipelining replication: peer="{Voter server-a37db21b-fa37-cb02-7d08-0e0fb48e698b a37db21b-fa37-cb02-7d08-0e0fb48e698b}" -2024-07-05T13:30:34.371Z [INFO] server-4daca39e-d423-537b-0066-2cfae97031d4: pipelining replication: peer="{Voter server-962bbec8-bdd6-c589-a7bb-bcddc12c8a72 962bbec8-bdd6-c589-a7bb-bcddc12c8a72}" -2024-07-05T13:30:35.376Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:34.371687862 +0000 UTC m=+75.141598537 (1 nodes), 637.001211ms from start of poll, 637.687187ms from cluster start. Timeout at 2024-07-05 13:30:35.376789 +0000 UTC m=+76.146699675, 1.005101138s after stability -2024-07-05T13:30:35.378Z [INFO] server-4daca39e-d423-537b-0066-2cfae97031d4: starting snapshot up to: index=12 -2024-07-05T13:30:35.378Z [INFO] creating new snapshot: path=/tmp/raft1513443526/snapshots/2-12-1720186235378.tmp -2024-07-05T13:30:35.379Z [INFO] server-4daca39e-d423-537b-0066-2cfae97031d4: snapshot complete up to: index=12 -2024-07-05T13:30:35.383Z [INFO] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20eb897f-dc0b-4eeb-ea92-36d298d657e9 Address:20eb897f-dc0b-4eeb-ea92-36d298d657e9} {Suffrage:Voter ID:server-c432bd9a-c91e-db69-59b4-6447520e3bfa Address:c432bd9a-c91e-db69-59b4-6447520e3bfa} {Suffrage:Voter ID:server-37d9dbd4-551f-960f-7040-7070fb8ca747 Address:37d9dbd4-551f-960f-7040-7070fb8ca747}]" -2024-07-05T13:30:35.384Z [INFO] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: entering follower state: follower="Node at 20eb897f-dc0b-4eeb-ea92-36d298d657e9 [Follower]" leader-address= leader-id= -2024-07-05T13:30:35.385Z [INFO] server-c432bd9a-c91e-db69-59b4-6447520e3bfa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20eb897f-dc0b-4eeb-ea92-36d298d657e9 Address:20eb897f-dc0b-4eeb-ea92-36d298d657e9} {Suffrage:Voter ID:server-c432bd9a-c91e-db69-59b4-6447520e3bfa Address:c432bd9a-c91e-db69-59b4-6447520e3bfa} {Suffrage:Voter ID:server-37d9dbd4-551f-960f-7040-7070fb8ca747 Address:37d9dbd4-551f-960f-7040-7070fb8ca747}]" -2024-07-05T13:30:35.385Z [INFO] server-c432bd9a-c91e-db69-59b4-6447520e3bfa: entering follower state: follower="Node at c432bd9a-c91e-db69-59b4-6447520e3bfa [Follower]" leader-address= leader-id= -2024-07-05T13:30:35.385Z [INFO] server-37d9dbd4-551f-960f-7040-7070fb8ca747: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20eb897f-dc0b-4eeb-ea92-36d298d657e9 Address:20eb897f-dc0b-4eeb-ea92-36d298d657e9} {Suffrage:Voter ID:server-c432bd9a-c91e-db69-59b4-6447520e3bfa Address:c432bd9a-c91e-db69-59b4-6447520e3bfa} {Suffrage:Voter ID:server-37d9dbd4-551f-960f-7040-7070fb8ca747 Address:37d9dbd4-551f-960f-7040-7070fb8ca747}]" -2024-07-05T13:30:35.385Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:35.385Z [INFO] server-37d9dbd4-551f-960f-7040-7070fb8ca747: entering follower state: follower="Node at 37d9dbd4-551f-960f-7040-7070fb8ca747 [Follower]" leader-address= leader-id= -2024-07-05T13:30:35.972Z [WARN] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:35.972Z [INFO] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: entering candidate state: node="Node at 20eb897f-dc0b-4eeb-ea92-36d298d657e9 [Candidate]" term=2 -2024-07-05T13:30:35.972Z [INFO] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: election won: term=2 tally=2 -2024-07-05T13:30:35.972Z [INFO] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: entering leader state: leader="Node at 20eb897f-dc0b-4eeb-ea92-36d298d657e9 [Leader]" -2024-07-05T13:30:35.972Z [INFO] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: added peer, starting replication: peer=server-c432bd9a-c91e-db69-59b4-6447520e3bfa -2024-07-05T13:30:35.972Z [INFO] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: added peer, starting replication: peer=server-37d9dbd4-551f-960f-7040-7070fb8ca747 -2024-07-05T13:30:35.972Z [INFO] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: pipelining replication: peer="{Voter server-37d9dbd4-551f-960f-7040-7070fb8ca747 37d9dbd4-551f-960f-7040-7070fb8ca747}" -2024-07-05T13:30:35.972Z [INFO] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: pipelining replication: peer="{Voter server-c432bd9a-c91e-db69-59b4-6447520e3bfa c432bd9a-c91e-db69-59b4-6447520e3bfa}" -2024-07-05T13:30:36.980Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:35.97308944 +0000 UTC m=+76.743000115 (1 nodes), 587.725141ms from start of poll, 589.494039ms from cluster start. Timeout at 2024-07-05 13:30:36.979945836 +0000 UTC m=+77.749856551, 1.006856436s after stability -2024-07-05T13:30:36.980Z [INFO] creating new snapshot: path=/tmp/raft2119847465/snapshots/2-1013-1720186236980.tmp -2024-07-05T13:30:36.981Z [INFO] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: copied to local snapshot: bytes=71 -2024-07-05T13:30:36.983Z [INFO] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: snapshot restore progress: id=2-1013-1720186236980 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:36.983Z [INFO] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: restored user snapshot: index=1013 -2024-07-05T13:30:36.983Z [ERROR] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: failed to get log: index=3 error="log not found" -2024-07-05T13:30:36.983Z [INFO] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: aborting pipeline replication: peer="{Voter server-c432bd9a-c91e-db69-59b4-6447520e3bfa c432bd9a-c91e-db69-59b4-6447520e3bfa}" -2024-07-05T13:30:36.983Z [ERROR] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: failed to get log: index=3 error="log not found" -2024-07-05T13:30:36.983Z [INFO] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: aborting pipeline replication: peer="{Voter server-37d9dbd4-551f-960f-7040-7070fb8ca747 37d9dbd4-551f-960f-7040-7070fb8ca747}" -2024-07-05T13:30:36.989Z [ERROR] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: failed to get log: index=3 error="log not found" -2024-07-05T13:30:36.989Z [INFO] creating new snapshot: path=/tmp/raft3125680198/snapshots/2-1013-1720186236989.tmp -2024-07-05T13:30:36.990Z [INFO] server-c432bd9a-c91e-db69-59b4-6447520e3bfa: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:36.991Z [INFO] server-c432bd9a-c91e-db69-59b4-6447520e3bfa: copied to local snapshot: bytes=71 -2024-07-05T13:30:36.992Z [INFO] server-c432bd9a-c91e-db69-59b4-6447520e3bfa: snapshot restore progress: id=2-1013-1720186236989 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:36.992Z [INFO] server-c432bd9a-c91e-db69-59b4-6447520e3bfa: Installed remote snapshot -2024-07-05T13:30:36.992Z [INFO] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: pipelining replication: peer="{Voter server-c432bd9a-c91e-db69-59b4-6447520e3bfa c432bd9a-c91e-db69-59b4-6447520e3bfa}" -2024-07-05T13:30:36.993Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:36.993Z [ERROR] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: failed to get log: index=3 error="log not found" -2024-07-05T13:30:36.993Z [INFO] creating new snapshot: path=/tmp/raft3964517885/snapshots/2-1013-1720186236993.tmp -2024-07-05T13:30:36.994Z [INFO] server-37d9dbd4-551f-960f-7040-7070fb8ca747: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:36.994Z [INFO] server-37d9dbd4-551f-960f-7040-7070fb8ca747: copied to local snapshot: bytes=71 -2024-07-05T13:30:36.997Z [INFO] server-37d9dbd4-551f-960f-7040-7070fb8ca747: snapshot restore progress: id=2-1013-1720186236993 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:36.997Z [INFO] server-37d9dbd4-551f-960f-7040-7070fb8ca747: Installed remote snapshot -2024-07-05T13:30:36.997Z [INFO] server-20eb897f-dc0b-4eeb-ea92-36d298d657e9: pipelining replication: peer="{Voter server-37d9dbd4-551f-960f-7040-7070fb8ca747 37d9dbd4-551f-960f-7040-7070fb8ca747}" -2024-07-05T13:30:37.998Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:36.993025857 +0000 UTC m=+77.762936532 (2 nodes), 1.92µs from start of poll, 1.609430456s from cluster start. Timeout at 2024-07-05 13:30:37.998391185 +0000 UTC m=+78.768301860, 1.005365328s after stability -2024-07-05T13:30:38.027Z [INFO] server-4daca39e-d423-537b-0066-2cfae97031d4: aborting pipeline replication: peer="{Voter server-962bbec8-bdd6-c589-a7bb-bcddc12c8a72 962bbec8-bdd6-c589-a7bb-bcddc12c8a72}" -2024-07-05T13:30:38.027Z [INFO] server-4daca39e-d423-537b-0066-2cfae97031d4: aborting pipeline replication: peer="{Voter server-a37db21b-fa37-cb02-7d08-0e0fb48e698b a37db21b-fa37-cb02-7d08-0e0fb48e698b}" +2024-07-05T13:36:51.059Z [INFO] server-f973b4cd-10b8-d60e-ccf8-14915d22dbb1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f973b4cd-10b8-d60e-ccf8-14915d22dbb1 Address:f973b4cd-10b8-d60e-ccf8-14915d22dbb1} {Suffrage:Voter ID:server-a40111f0-df42-fca4-435c-87d06089af8a Address:a40111f0-df42-fca4-435c-87d06089af8a} {Suffrage:Voter ID:server-84916c25-77f7-db17-01cc-10c7ee262ca9 Address:84916c25-77f7-db17-01cc-10c7ee262ca9}]" +2024-07-05T13:36:51.060Z [INFO] server-a40111f0-df42-fca4-435c-87d06089af8a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f973b4cd-10b8-d60e-ccf8-14915d22dbb1 Address:f973b4cd-10b8-d60e-ccf8-14915d22dbb1} {Suffrage:Voter ID:server-a40111f0-df42-fca4-435c-87d06089af8a Address:a40111f0-df42-fca4-435c-87d06089af8a} {Suffrage:Voter ID:server-84916c25-77f7-db17-01cc-10c7ee262ca9 Address:84916c25-77f7-db17-01cc-10c7ee262ca9}]" +2024-07-05T13:36:51.060Z [INFO] server-f973b4cd-10b8-d60e-ccf8-14915d22dbb1: entering follower state: follower="Node at f973b4cd-10b8-d60e-ccf8-14915d22dbb1 [Follower]" leader-address= leader-id= +2024-07-05T13:36:51.061Z [INFO] server-a40111f0-df42-fca4-435c-87d06089af8a: entering follower state: follower="Node at a40111f0-df42-fca4-435c-87d06089af8a [Follower]" leader-address= leader-id= +2024-07-05T13:36:51.061Z [INFO] server-84916c25-77f7-db17-01cc-10c7ee262ca9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f973b4cd-10b8-d60e-ccf8-14915d22dbb1 Address:f973b4cd-10b8-d60e-ccf8-14915d22dbb1} {Suffrage:Voter ID:server-a40111f0-df42-fca4-435c-87d06089af8a Address:a40111f0-df42-fca4-435c-87d06089af8a} {Suffrage:Voter ID:server-84916c25-77f7-db17-01cc-10c7ee262ca9 Address:84916c25-77f7-db17-01cc-10c7ee262ca9}]" +2024-07-05T13:36:51.061Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:51.062Z [INFO] server-84916c25-77f7-db17-01cc-10c7ee262ca9: entering follower state: follower="Node at 84916c25-77f7-db17-01cc-10c7ee262ca9 [Follower]" leader-address= leader-id= +2024-07-05T13:36:51.633Z [WARN] server-a40111f0-df42-fca4-435c-87d06089af8a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:51.633Z [INFO] server-a40111f0-df42-fca4-435c-87d06089af8a: entering candidate state: node="Node at a40111f0-df42-fca4-435c-87d06089af8a [Candidate]" term=2 +2024-07-05T13:36:51.634Z [INFO] server-a40111f0-df42-fca4-435c-87d06089af8a: election won: term=2 tally=2 +2024-07-05T13:36:51.634Z [INFO] server-a40111f0-df42-fca4-435c-87d06089af8a: entering leader state: leader="Node at a40111f0-df42-fca4-435c-87d06089af8a [Leader]" +2024-07-05T13:36:51.634Z [INFO] server-a40111f0-df42-fca4-435c-87d06089af8a: added peer, starting replication: peer=server-f973b4cd-10b8-d60e-ccf8-14915d22dbb1 +2024-07-05T13:36:51.634Z [INFO] server-a40111f0-df42-fca4-435c-87d06089af8a: added peer, starting replication: peer=server-84916c25-77f7-db17-01cc-10c7ee262ca9 +2024-07-05T13:36:51.635Z [INFO] server-a40111f0-df42-fca4-435c-87d06089af8a: pipelining replication: peer="{Voter server-f973b4cd-10b8-d60e-ccf8-14915d22dbb1 f973b4cd-10b8-d60e-ccf8-14915d22dbb1}" +2024-07-05T13:36:51.635Z [INFO] server-a40111f0-df42-fca4-435c-87d06089af8a: pipelining replication: peer="{Voter server-84916c25-77f7-db17-01cc-10c7ee262ca9 84916c25-77f7-db17-01cc-10c7ee262ca9}" +2024-07-05T13:36:52.640Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:51.634814293 +0000 UTC m=+77.630736037 (1 nodes), 572.809165ms from start of poll, 575.356826ms from cluster start. Timeout at 2024-07-05 13:36:52.639835231 +0000 UTC m=+78.635756682, 1.005020645s after stability +2024-07-05T13:36:52.645Z [INFO] server-a40111f0-df42-fca4-435c-87d06089af8a: starting snapshot up to: index=12 +2024-07-05T13:36:52.646Z [INFO] creating new snapshot: path=/tmp/raft2152284601/snapshots/2-12-1720186612646.tmp +2024-07-05T13:36:52.654Z [INFO] server-a40111f0-df42-fca4-435c-87d06089af8a: snapshot complete up to: index=12 +2024-07-05T13:36:52.663Z [INFO] server-9df449b3-03b7-1ec9-1bf3-9481571d440e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9df449b3-03b7-1ec9-1bf3-9481571d440e Address:9df449b3-03b7-1ec9-1bf3-9481571d440e} {Suffrage:Voter ID:server-f53df544-46e0-2b4d-cffb-e3bc643e4401 Address:f53df544-46e0-2b4d-cffb-e3bc643e4401} {Suffrage:Voter ID:server-ba8a1951-6c3c-356a-58d5-30e2727a3836 Address:ba8a1951-6c3c-356a-58d5-30e2727a3836}]" +2024-07-05T13:36:52.663Z [INFO] server-9df449b3-03b7-1ec9-1bf3-9481571d440e: entering follower state: follower="Node at 9df449b3-03b7-1ec9-1bf3-9481571d440e [Follower]" leader-address= leader-id= +2024-07-05T13:36:52.663Z [INFO] server-f53df544-46e0-2b4d-cffb-e3bc643e4401: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9df449b3-03b7-1ec9-1bf3-9481571d440e Address:9df449b3-03b7-1ec9-1bf3-9481571d440e} {Suffrage:Voter ID:server-f53df544-46e0-2b4d-cffb-e3bc643e4401 Address:f53df544-46e0-2b4d-cffb-e3bc643e4401} {Suffrage:Voter ID:server-ba8a1951-6c3c-356a-58d5-30e2727a3836 Address:ba8a1951-6c3c-356a-58d5-30e2727a3836}]" +2024-07-05T13:36:52.664Z [INFO] server-f53df544-46e0-2b4d-cffb-e3bc643e4401: entering follower state: follower="Node at f53df544-46e0-2b4d-cffb-e3bc643e4401 [Follower]" leader-address= leader-id= +2024-07-05T13:36:52.664Z [INFO] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9df449b3-03b7-1ec9-1bf3-9481571d440e Address:9df449b3-03b7-1ec9-1bf3-9481571d440e} {Suffrage:Voter ID:server-f53df544-46e0-2b4d-cffb-e3bc643e4401 Address:f53df544-46e0-2b4d-cffb-e3bc643e4401} {Suffrage:Voter ID:server-ba8a1951-6c3c-356a-58d5-30e2727a3836 Address:ba8a1951-6c3c-356a-58d5-30e2727a3836}]" +2024-07-05T13:36:52.664Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:52.664Z [INFO] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: entering follower state: follower="Node at ba8a1951-6c3c-356a-58d5-30e2727a3836 [Follower]" leader-address= leader-id= +2024-07-05T13:36:53.271Z [WARN] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:53.271Z [INFO] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: entering candidate state: node="Node at ba8a1951-6c3c-356a-58d5-30e2727a3836 [Candidate]" term=2 +2024-07-05T13:36:53.272Z [INFO] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: election won: term=2 tally=2 +2024-07-05T13:36:53.272Z [INFO] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: entering leader state: leader="Node at ba8a1951-6c3c-356a-58d5-30e2727a3836 [Leader]" +2024-07-05T13:36:53.272Z [INFO] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: added peer, starting replication: peer=server-9df449b3-03b7-1ec9-1bf3-9481571d440e +2024-07-05T13:36:53.272Z [INFO] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: added peer, starting replication: peer=server-f53df544-46e0-2b4d-cffb-e3bc643e4401 +2024-07-05T13:36:53.273Z [INFO] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: pipelining replication: peer="{Voter server-f53df544-46e0-2b4d-cffb-e3bc643e4401 f53df544-46e0-2b4d-cffb-e3bc643e4401}" +2024-07-05T13:36:53.274Z [INFO] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: pipelining replication: peer="{Voter server-9df449b3-03b7-1ec9-1bf3-9481571d440e 9df449b3-03b7-1ec9-1bf3-9481571d440e}" +2024-07-05T13:36:54.278Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:53.273360994 +0000 UTC m=+79.269282446 (1 nodes), 608.783257ms from start of poll, 610.833636ms from cluster start. Timeout at 2024-07-05 13:36:54.27840264 +0000 UTC m=+80.274323800, 1.005041354s after stability +2024-07-05T13:36:54.280Z [INFO] creating new snapshot: path=/tmp/raft2074055966/snapshots/2-1013-1720186614280.tmp +2024-07-05T13:36:54.289Z [INFO] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: copied to local snapshot: bytes=71 +2024-07-05T13:36:54.297Z [INFO] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: snapshot restore progress: id=2-1013-1720186614280 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:54.297Z [INFO] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: restored user snapshot: index=1013 +2024-07-05T13:36:54.297Z [ERROR] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: failed to get log: index=3 error="log not found" +2024-07-05T13:36:54.297Z [INFO] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: aborting pipeline replication: peer="{Voter server-9df449b3-03b7-1ec9-1bf3-9481571d440e 9df449b3-03b7-1ec9-1bf3-9481571d440e}" +2024-07-05T13:36:54.298Z [ERROR] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: failed to get log: index=3 error="log not found" +2024-07-05T13:36:54.298Z [INFO] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: aborting pipeline replication: peer="{Voter server-f53df544-46e0-2b4d-cffb-e3bc643e4401 f53df544-46e0-2b4d-cffb-e3bc643e4401}" +2024-07-05T13:36:54.303Z [ERROR] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: failed to get log: index=3 error="log not found" +2024-07-05T13:36:54.303Z [ERROR] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: failed to get log: index=3 error="log not found" +2024-07-05T13:36:54.304Z [INFO] creating new snapshot: path=/tmp/raft1190150052/snapshots/2-1013-1720186614304.tmp +2024-07-05T13:36:54.305Z [INFO] creating new snapshot: path=/tmp/raft1592272379/snapshots/2-1013-1720186614305.tmp +2024-07-05T13:36:54.306Z [INFO] server-9df449b3-03b7-1ec9-1bf3-9481571d440e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:54.306Z [INFO] server-f53df544-46e0-2b4d-cffb-e3bc643e4401: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:54.308Z [INFO] server-9df449b3-03b7-1ec9-1bf3-9481571d440e: copied to local snapshot: bytes=71 +2024-07-05T13:36:54.308Z [INFO] server-f53df544-46e0-2b4d-cffb-e3bc643e4401: copied to local snapshot: bytes=71 +2024-07-05T13:36:54.309Z [INFO] server-f53df544-46e0-2b4d-cffb-e3bc643e4401: snapshot restore progress: id=2-1013-1720186614304 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:54.309Z [INFO] server-9df449b3-03b7-1ec9-1bf3-9481571d440e: snapshot restore progress: id=2-1013-1720186614305 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:54.309Z [INFO] server-f53df544-46e0-2b4d-cffb-e3bc643e4401: Installed remote snapshot +2024-07-05T13:36:54.310Z [INFO] server-9df449b3-03b7-1ec9-1bf3-9481571d440e: Installed remote snapshot +2024-07-05T13:36:54.310Z [WARN] server-9df449b3-03b7-1ec9-1bf3-9481571d440e: skipping application of old log: index=1013 +2024-07-05T13:36:54.310Z [INFO] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: pipelining replication: peer="{Voter server-f53df544-46e0-2b4d-cffb-e3bc643e4401 f53df544-46e0-2b4d-cffb-e3bc643e4401}" +2024-07-05T13:36:54.310Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:36:54.311Z [INFO] server-ba8a1951-6c3c-356a-58d5-30e2727a3836: pipelining replication: peer="{Voter server-9df449b3-03b7-1ec9-1bf3-9481571d440e 9df449b3-03b7-1ec9-1bf3-9481571d440e}" +2024-07-05T13:36:55.316Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:36:54.310867464 +0000 UTC m=+80.306788624 (2 nodes), 5.833µs from start of poll, 1.648339814s from cluster start. Timeout at 2024-07-05 13:36:55.315953734 +0000 UTC m=+81.311875185, 1.005086561s after stability +2024-07-05T13:36:55.331Z [INFO] server-a40111f0-df42-fca4-435c-87d06089af8a: aborting pipeline replication: peer="{Voter server-84916c25-77f7-db17-01cc-10c7ee262ca9 84916c25-77f7-db17-01cc-10c7ee262ca9}" +2024-07-05T13:36:55.331Z [INFO] server-a40111f0-df42-fca4-435c-87d06089af8a: aborting pipeline replication: peer="{Voter server-f973b4cd-10b8-d60e-ccf8-14915d22dbb1 f973b4cd-10b8-d60e-ccf8-14915d22dbb1}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true -2024-07-05T13:30:38.036Z [INFO] server-e78e91ba-0646-8eaf-35e8-6ecdad491b38: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e78e91ba-0646-8eaf-35e8-6ecdad491b38 Address:e78e91ba-0646-8eaf-35e8-6ecdad491b38} {Suffrage:Voter ID:server-4b4fa385-06c0-8977-0b02-d891fdf41a29 Address:4b4fa385-06c0-8977-0b02-d891fdf41a29} {Suffrage:Voter ID:server-925bfaa7-f24c-296d-bded-120eb5dee11a Address:925bfaa7-f24c-296d-bded-120eb5dee11a}]" -2024-07-05T13:30:38.036Z [INFO] server-e78e91ba-0646-8eaf-35e8-6ecdad491b38: entering follower state: follower="Node at e78e91ba-0646-8eaf-35e8-6ecdad491b38 [Follower]" leader-address= leader-id= -2024-07-05T13:30:38.036Z [INFO] server-4b4fa385-06c0-8977-0b02-d891fdf41a29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e78e91ba-0646-8eaf-35e8-6ecdad491b38 Address:e78e91ba-0646-8eaf-35e8-6ecdad491b38} {Suffrage:Voter ID:server-4b4fa385-06c0-8977-0b02-d891fdf41a29 Address:4b4fa385-06c0-8977-0b02-d891fdf41a29} {Suffrage:Voter ID:server-925bfaa7-f24c-296d-bded-120eb5dee11a Address:925bfaa7-f24c-296d-bded-120eb5dee11a}]" -2024-07-05T13:30:38.036Z [INFO] server-925bfaa7-f24c-296d-bded-120eb5dee11a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e78e91ba-0646-8eaf-35e8-6ecdad491b38 Address:e78e91ba-0646-8eaf-35e8-6ecdad491b38} {Suffrage:Voter ID:server-4b4fa385-06c0-8977-0b02-d891fdf41a29 Address:4b4fa385-06c0-8977-0b02-d891fdf41a29} {Suffrage:Voter ID:server-925bfaa7-f24c-296d-bded-120eb5dee11a Address:925bfaa7-f24c-296d-bded-120eb5dee11a}]" -2024-07-05T13:30:38.036Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:38.036Z [INFO] server-4b4fa385-06c0-8977-0b02-d891fdf41a29: entering follower state: follower="Node at 4b4fa385-06c0-8977-0b02-d891fdf41a29 [Follower]" leader-address= leader-id= -2024-07-05T13:30:38.036Z [INFO] server-925bfaa7-f24c-296d-bded-120eb5dee11a: entering follower state: follower="Node at 925bfaa7-f24c-296d-bded-120eb5dee11a [Follower]" leader-address= leader-id= -2024-07-05T13:30:38.552Z [WARN] server-4b4fa385-06c0-8977-0b02-d891fdf41a29: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:38.552Z [INFO] server-4b4fa385-06c0-8977-0b02-d891fdf41a29: entering candidate state: node="Node at 4b4fa385-06c0-8977-0b02-d891fdf41a29 [Candidate]" term=2 -2024-07-05T13:30:38.552Z [INFO] server-4b4fa385-06c0-8977-0b02-d891fdf41a29: election won: term=2 tally=2 -2024-07-05T13:30:38.552Z [INFO] server-4b4fa385-06c0-8977-0b02-d891fdf41a29: entering leader state: leader="Node at 4b4fa385-06c0-8977-0b02-d891fdf41a29 [Leader]" -2024-07-05T13:30:38.552Z [INFO] server-4b4fa385-06c0-8977-0b02-d891fdf41a29: added peer, starting replication: peer=server-e78e91ba-0646-8eaf-35e8-6ecdad491b38 -2024-07-05T13:30:38.552Z [INFO] server-4b4fa385-06c0-8977-0b02-d891fdf41a29: added peer, starting replication: peer=server-925bfaa7-f24c-296d-bded-120eb5dee11a -2024-07-05T13:30:38.552Z [INFO] server-4b4fa385-06c0-8977-0b02-d891fdf41a29: pipelining replication: peer="{Voter server-925bfaa7-f24c-296d-bded-120eb5dee11a 925bfaa7-f24c-296d-bded-120eb5dee11a}" -2024-07-05T13:30:38.553Z [INFO] server-4b4fa385-06c0-8977-0b02-d891fdf41a29: pipelining replication: peer="{Voter server-e78e91ba-0646-8eaf-35e8-6ecdad491b38 e78e91ba-0646-8eaf-35e8-6ecdad491b38}" -2024-07-05T13:30:39.558Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:38.552856092 +0000 UTC m=+79.322766767 (1 nodes), 516.058534ms from start of poll, 517.0279ms from cluster start. Timeout at 2024-07-05 13:30:39.558434333 +0000 UTC m=+80.328345008, 1.005578241s after stability -2024-07-05T13:30:39.560Z [INFO] server-4b4fa385-06c0-8977-0b02-d891fdf41a29: starting snapshot up to: index=12 -2024-07-05T13:30:39.560Z [INFO] creating new snapshot: path=/tmp/raft3739274916/snapshots/2-12-1720186239560.tmp -2024-07-05T13:30:39.568Z [INFO] server-4b4fa385-06c0-8977-0b02-d891fdf41a29: snapshot complete up to: index=12 -2024-07-05T13:30:39.584Z [INFO] server-d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136 Address:d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136} {Suffrage:Voter ID:server-0f8e6e06-1a5a-d918-6cb8-250e55565175 Address:0f8e6e06-1a5a-d918-6cb8-250e55565175} {Suffrage:Voter ID:server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f Address:c6d55dc6-c3d8-a570-d23c-e09bcd2c834f}]" -2024-07-05T13:30:39.584Z [INFO] server-d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136: entering follower state: follower="Node at d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136 [Follower]" leader-address= leader-id= -2024-07-05T13:30:39.584Z [INFO] server-0f8e6e06-1a5a-d918-6cb8-250e55565175: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136 Address:d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136} {Suffrage:Voter ID:server-0f8e6e06-1a5a-d918-6cb8-250e55565175 Address:0f8e6e06-1a5a-d918-6cb8-250e55565175} {Suffrage:Voter ID:server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f Address:c6d55dc6-c3d8-a570-d23c-e09bcd2c834f}]" -2024-07-05T13:30:39.584Z [INFO] server-0f8e6e06-1a5a-d918-6cb8-250e55565175: entering follower state: follower="Node at 0f8e6e06-1a5a-d918-6cb8-250e55565175 [Follower]" leader-address= leader-id= -2024-07-05T13:30:39.584Z [INFO] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136 Address:d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136} {Suffrage:Voter ID:server-0f8e6e06-1a5a-d918-6cb8-250e55565175 Address:0f8e6e06-1a5a-d918-6cb8-250e55565175} {Suffrage:Voter ID:server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f Address:c6d55dc6-c3d8-a570-d23c-e09bcd2c834f}]" -2024-07-05T13:30:39.584Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:39.584Z [INFO] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: entering follower state: follower="Node at c6d55dc6-c3d8-a570-d23c-e09bcd2c834f [Follower]" leader-address= leader-id= -2024-07-05T13:30:40.180Z [WARN] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:40.180Z [INFO] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: entering candidate state: node="Node at c6d55dc6-c3d8-a570-d23c-e09bcd2c834f [Candidate]" term=2 -2024-07-05T13:30:40.180Z [INFO] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: election won: term=2 tally=2 -2024-07-05T13:30:40.180Z [INFO] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: entering leader state: leader="Node at c6d55dc6-c3d8-a570-d23c-e09bcd2c834f [Leader]" -2024-07-05T13:30:40.180Z [INFO] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: added peer, starting replication: peer=server-d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136 -2024-07-05T13:30:40.180Z [INFO] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: added peer, starting replication: peer=server-0f8e6e06-1a5a-d918-6cb8-250e55565175 -2024-07-05T13:30:40.180Z [INFO] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: pipelining replication: peer="{Voter server-d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136 d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136}" -2024-07-05T13:30:40.180Z [INFO] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: pipelining replication: peer="{Voter server-0f8e6e06-1a5a-d918-6cb8-250e55565175 0f8e6e06-1a5a-d918-6cb8-250e55565175}" -2024-07-05T13:30:41.186Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:40.18074606 +0000 UTC m=+80.950656735 (1 nodes), 595.784458ms from start of poll, 596.537391ms from cluster start. Timeout at 2024-07-05 13:30:41.186687488 +0000 UTC m=+81.956598163, 1.005941428s after stability -2024-07-05T13:30:41.187Z [INFO] creating new snapshot: path=/tmp/raft3770447140/snapshots/2-1013-1720186241187.tmp -2024-07-05T13:30:41.188Z [INFO] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: copied to local snapshot: bytes=71 -2024-07-05T13:30:41.189Z [INFO] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: snapshot restore progress: id=2-1013-1720186241187 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:41.189Z [INFO] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: removing all old logs from log store -2024-07-05T13:30:41.189Z [INFO] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: compacting logs: from=1 to=2 -2024-07-05T13:30:41.189Z [INFO] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: restored user snapshot: index=1013 -2024-07-05T13:30:41.189Z [ERROR] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: failed to get log: index=2 error="log not found" -2024-07-05T13:30:41.189Z [INFO] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: aborting pipeline replication: peer="{Voter server-0f8e6e06-1a5a-d918-6cb8-250e55565175 0f8e6e06-1a5a-d918-6cb8-250e55565175}" -2024-07-05T13:30:41.189Z [ERROR] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: failed to get log: index=2 error="log not found" -2024-07-05T13:30:41.189Z [INFO] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: aborting pipeline replication: peer="{Voter server-d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136 d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136}" -2024-07-05T13:30:41.195Z [ERROR] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: failed to get log: index=2 error="log not found" -2024-07-05T13:30:41.196Z [ERROR] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: failed to get log: index=2 error="log not found" -2024-07-05T13:30:41.197Z [INFO] creating new snapshot: path=/tmp/raft455340487/snapshots/2-1013-1720186241197.tmp -2024-07-05T13:30:41.197Z [INFO] creating new snapshot: path=/tmp/raft146614804/snapshots/2-1013-1720186241197.tmp -2024-07-05T13:30:41.197Z [INFO] server-d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:41.197Z [INFO] server-0f8e6e06-1a5a-d918-6cb8-250e55565175: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:41.198Z [INFO] server-d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136: copied to local snapshot: bytes=71 -2024-07-05T13:30:41.198Z [INFO] server-0f8e6e06-1a5a-d918-6cb8-250e55565175: copied to local snapshot: bytes=71 -2024-07-05T13:30:41.198Z [INFO] server-d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136: snapshot restore progress: id=2-1013-1720186241197 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:41.198Z [INFO] server-d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136: removing all old logs from log store -2024-07-05T13:30:41.198Z [INFO] server-d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136: compacting logs: from=1 to=2 -2024-07-05T13:30:41.198Z [INFO] server-d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136: Installed remote snapshot -2024-07-05T13:30:41.198Z [INFO] server-0f8e6e06-1a5a-d918-6cb8-250e55565175: snapshot restore progress: id=2-1013-1720186241197 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:41.198Z [INFO] server-0f8e6e06-1a5a-d918-6cb8-250e55565175: removing all old logs from log store -2024-07-05T13:30:41.198Z [INFO] server-0f8e6e06-1a5a-d918-6cb8-250e55565175: compacting logs: from=1 to=2 -2024-07-05T13:30:41.199Z [INFO] server-0f8e6e06-1a5a-d918-6cb8-250e55565175: Installed remote snapshot -2024-07-05T13:30:41.199Z [INFO] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: pipelining replication: peer="{Voter server-d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136 d0ac1a8c-ffd3-5bee-0fb1-8e8869ff0136}" -2024-07-05T13:30:41.199Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:41.199Z [INFO] server-c6d55dc6-c3d8-a570-d23c-e09bcd2c834f: pipelining replication: peer="{Voter server-0f8e6e06-1a5a-d918-6cb8-250e55565175 0f8e6e06-1a5a-d918-6cb8-250e55565175}" -2024-07-05T13:30:42.205Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:41.199282486 +0000 UTC m=+81.969193161 (2 nodes), 1.92µs from start of poll, 1.615073817s from cluster start. Timeout at 2024-07-05 13:30:42.204878806 +0000 UTC m=+82.974789521, 1.00559636s after stability -2024-07-05T13:30:42.217Z [INFO] server-4b4fa385-06c0-8977-0b02-d891fdf41a29: aborting pipeline replication: peer="{Voter server-925bfaa7-f24c-296d-bded-120eb5dee11a 925bfaa7-f24c-296d-bded-120eb5dee11a}" -2024-07-05T13:30:42.217Z [INFO] server-4b4fa385-06c0-8977-0b02-d891fdf41a29: aborting pipeline replication: peer="{Voter server-e78e91ba-0646-8eaf-35e8-6ecdad491b38 e78e91ba-0646-8eaf-35e8-6ecdad491b38}" +2024-07-05T13:36:55.344Z [INFO] server-9b15fc1c-eb9c-02cc-239e-3a802ff2d30f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b15fc1c-eb9c-02cc-239e-3a802ff2d30f Address:9b15fc1c-eb9c-02cc-239e-3a802ff2d30f} {Suffrage:Voter ID:server-285b0f21-70a5-08b0-6dc1-aa822e4cfb45 Address:285b0f21-70a5-08b0-6dc1-aa822e4cfb45} {Suffrage:Voter ID:server-071be13a-7f98-f7d3-2a5d-0f657c8dbeac Address:071be13a-7f98-f7d3-2a5d-0f657c8dbeac}]" +2024-07-05T13:36:55.345Z [INFO] server-285b0f21-70a5-08b0-6dc1-aa822e4cfb45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b15fc1c-eb9c-02cc-239e-3a802ff2d30f Address:9b15fc1c-eb9c-02cc-239e-3a802ff2d30f} {Suffrage:Voter ID:server-285b0f21-70a5-08b0-6dc1-aa822e4cfb45 Address:285b0f21-70a5-08b0-6dc1-aa822e4cfb45} {Suffrage:Voter ID:server-071be13a-7f98-f7d3-2a5d-0f657c8dbeac Address:071be13a-7f98-f7d3-2a5d-0f657c8dbeac}]" +2024-07-05T13:36:55.345Z [INFO] server-9b15fc1c-eb9c-02cc-239e-3a802ff2d30f: entering follower state: follower="Node at 9b15fc1c-eb9c-02cc-239e-3a802ff2d30f [Follower]" leader-address= leader-id= +2024-07-05T13:36:55.345Z [INFO] server-285b0f21-70a5-08b0-6dc1-aa822e4cfb45: entering follower state: follower="Node at 285b0f21-70a5-08b0-6dc1-aa822e4cfb45 [Follower]" leader-address= leader-id= +2024-07-05T13:36:55.345Z [INFO] server-071be13a-7f98-f7d3-2a5d-0f657c8dbeac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b15fc1c-eb9c-02cc-239e-3a802ff2d30f Address:9b15fc1c-eb9c-02cc-239e-3a802ff2d30f} {Suffrage:Voter ID:server-285b0f21-70a5-08b0-6dc1-aa822e4cfb45 Address:285b0f21-70a5-08b0-6dc1-aa822e4cfb45} {Suffrage:Voter ID:server-071be13a-7f98-f7d3-2a5d-0f657c8dbeac Address:071be13a-7f98-f7d3-2a5d-0f657c8dbeac}]" +2024-07-05T13:36:55.346Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:55.346Z [INFO] server-071be13a-7f98-f7d3-2a5d-0f657c8dbeac: entering follower state: follower="Node at 071be13a-7f98-f7d3-2a5d-0f657c8dbeac [Follower]" leader-address= leader-id= +2024-07-05T13:36:55.861Z [WARN] server-071be13a-7f98-f7d3-2a5d-0f657c8dbeac: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:55.862Z [INFO] server-071be13a-7f98-f7d3-2a5d-0f657c8dbeac: entering candidate state: node="Node at 071be13a-7f98-f7d3-2a5d-0f657c8dbeac [Candidate]" term=2 +2024-07-05T13:36:55.862Z [INFO] server-071be13a-7f98-f7d3-2a5d-0f657c8dbeac: election won: term=2 tally=2 +2024-07-05T13:36:55.862Z [INFO] server-071be13a-7f98-f7d3-2a5d-0f657c8dbeac: entering leader state: leader="Node at 071be13a-7f98-f7d3-2a5d-0f657c8dbeac [Leader]" +2024-07-05T13:36:55.862Z [INFO] server-071be13a-7f98-f7d3-2a5d-0f657c8dbeac: added peer, starting replication: peer=server-9b15fc1c-eb9c-02cc-239e-3a802ff2d30f +2024-07-05T13:36:55.863Z [INFO] server-071be13a-7f98-f7d3-2a5d-0f657c8dbeac: added peer, starting replication: peer=server-285b0f21-70a5-08b0-6dc1-aa822e4cfb45 +2024-07-05T13:36:55.864Z [INFO] server-071be13a-7f98-f7d3-2a5d-0f657c8dbeac: pipelining replication: peer="{Voter server-9b15fc1c-eb9c-02cc-239e-3a802ff2d30f 9b15fc1c-eb9c-02cc-239e-3a802ff2d30f}" +2024-07-05T13:36:55.865Z [INFO] server-071be13a-7f98-f7d3-2a5d-0f657c8dbeac: pipelining replication: peer="{Voter server-285b0f21-70a5-08b0-6dc1-aa822e4cfb45 285b0f21-70a5-08b0-6dc1-aa822e4cfb45}" +2024-07-05T13:36:56.869Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:55.864125098 +0000 UTC m=+81.860046550 (1 nodes), 517.867168ms from start of poll, 520.134252ms from cluster start. Timeout at 2024-07-05 13:36:56.869265325 +0000 UTC m=+82.865186485, 1.005139935s after stability +2024-07-05T13:36:56.872Z [INFO] server-071be13a-7f98-f7d3-2a5d-0f657c8dbeac: starting snapshot up to: index=12 +2024-07-05T13:36:56.873Z [INFO] creating new snapshot: path=/tmp/raft3160414745/snapshots/2-12-1720186616873.tmp +2024-07-05T13:36:56.876Z [INFO] server-071be13a-7f98-f7d3-2a5d-0f657c8dbeac: snapshot complete up to: index=12 +2024-07-05T13:36:56.900Z [INFO] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f Address:07db59d4-a90b-60b3-80aa-89a59e6b3d5f} {Suffrage:Voter ID:server-33a6508d-5612-e9c8-740b-dd7cb40e9784 Address:33a6508d-5612-e9c8-740b-dd7cb40e9784} {Suffrage:Voter ID:server-c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa Address:c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa}]" +2024-07-05T13:36:56.901Z [INFO] server-33a6508d-5612-e9c8-740b-dd7cb40e9784: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f Address:07db59d4-a90b-60b3-80aa-89a59e6b3d5f} {Suffrage:Voter ID:server-33a6508d-5612-e9c8-740b-dd7cb40e9784 Address:33a6508d-5612-e9c8-740b-dd7cb40e9784} {Suffrage:Voter ID:server-c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa Address:c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa}]" +2024-07-05T13:36:56.902Z [INFO] server-c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f Address:07db59d4-a90b-60b3-80aa-89a59e6b3d5f} {Suffrage:Voter ID:server-33a6508d-5612-e9c8-740b-dd7cb40e9784 Address:33a6508d-5612-e9c8-740b-dd7cb40e9784} {Suffrage:Voter ID:server-c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa Address:c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa}]" +2024-07-05T13:36:56.903Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:56.903Z [INFO] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: entering follower state: follower="Node at 07db59d4-a90b-60b3-80aa-89a59e6b3d5f [Follower]" leader-address= leader-id= +2024-07-05T13:36:56.903Z [INFO] server-33a6508d-5612-e9c8-740b-dd7cb40e9784: entering follower state: follower="Node at 33a6508d-5612-e9c8-740b-dd7cb40e9784 [Follower]" leader-address= leader-id= +2024-07-05T13:36:56.903Z [INFO] server-c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa: entering follower state: follower="Node at c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa [Follower]" leader-address= leader-id= +2024-07-05T13:36:57.473Z [WARN] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:36:57.473Z [INFO] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: entering candidate state: node="Node at 07db59d4-a90b-60b3-80aa-89a59e6b3d5f [Candidate]" term=2 +2024-07-05T13:36:57.474Z [INFO] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: election won: term=2 tally=2 +2024-07-05T13:36:57.474Z [INFO] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: entering leader state: leader="Node at 07db59d4-a90b-60b3-80aa-89a59e6b3d5f [Leader]" +2024-07-05T13:36:57.474Z [INFO] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: added peer, starting replication: peer=server-33a6508d-5612-e9c8-740b-dd7cb40e9784 +2024-07-05T13:36:57.474Z [INFO] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: added peer, starting replication: peer=server-c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa +2024-07-05T13:36:57.475Z [INFO] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: pipelining replication: peer="{Voter server-33a6508d-5612-e9c8-740b-dd7cb40e9784 33a6508d-5612-e9c8-740b-dd7cb40e9784}" +2024-07-05T13:36:57.475Z [INFO] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: pipelining replication: peer="{Voter server-c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa}" +2024-07-05T13:36:58.480Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:36:57.474861016 +0000 UTC m=+83.470782468 (1 nodes), 571.748393ms from start of poll, 574.616882ms from cluster start. Timeout at 2024-07-05 13:36:58.48007445 +0000 UTC m=+84.475995319, 1.005212851s after stability +2024-07-05T13:36:58.491Z [INFO] creating new snapshot: path=/tmp/raft3685958210/snapshots/2-1013-1720186618490.tmp +2024-07-05T13:36:58.495Z [INFO] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: copied to local snapshot: bytes=71 +2024-07-05T13:36:58.496Z [INFO] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: snapshot restore progress: id=2-1013-1720186618490 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:58.497Z [INFO] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: removing all old logs from log store +2024-07-05T13:36:58.497Z [INFO] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: compacting logs: from=1 to=2 +2024-07-05T13:36:58.497Z [INFO] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: restored user snapshot: index=1013 +2024-07-05T13:36:58.498Z [ERROR] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: failed to get log: index=2 error="log not found" +2024-07-05T13:36:58.498Z [INFO] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: aborting pipeline replication: peer="{Voter server-33a6508d-5612-e9c8-740b-dd7cb40e9784 33a6508d-5612-e9c8-740b-dd7cb40e9784}" +2024-07-05T13:36:58.498Z [ERROR] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: failed to get log: index=2 error="log not found" +2024-07-05T13:36:58.498Z [INFO] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: aborting pipeline replication: peer="{Voter server-c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa}" +2024-07-05T13:36:58.505Z [ERROR] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: failed to get log: index=2 error="log not found" +2024-07-05T13:36:58.507Z [INFO] creating new snapshot: path=/tmp/raft2277277869/snapshots/2-1013-1720186618507.tmp +2024-07-05T13:36:58.507Z [ERROR] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: failed to get log: index=2 error="log not found" +2024-07-05T13:36:58.509Z [INFO] creating new snapshot: path=/tmp/raft509311995/snapshots/2-1013-1720186618509.tmp +2024-07-05T13:36:58.509Z [INFO] server-33a6508d-5612-e9c8-740b-dd7cb40e9784: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:58.510Z [INFO] server-c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:58.511Z [INFO] server-33a6508d-5612-e9c8-740b-dd7cb40e9784: copied to local snapshot: bytes=71 +2024-07-05T13:36:58.511Z [INFO] server-c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa: copied to local snapshot: bytes=71 +2024-07-05T13:36:58.512Z [INFO] server-33a6508d-5612-e9c8-740b-dd7cb40e9784: snapshot restore progress: id=2-1013-1720186618507 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:58.512Z [INFO] server-33a6508d-5612-e9c8-740b-dd7cb40e9784: removing all old logs from log store +2024-07-05T13:36:58.512Z [INFO] server-33a6508d-5612-e9c8-740b-dd7cb40e9784: compacting logs: from=1 to=2 +2024-07-05T13:36:58.512Z [INFO] server-33a6508d-5612-e9c8-740b-dd7cb40e9784: Installed remote snapshot +2024-07-05T13:36:58.513Z [INFO] server-c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa: snapshot restore progress: id=2-1013-1720186618509 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:36:58.513Z [INFO] server-c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa: removing all old logs from log store +2024-07-05T13:36:58.513Z [INFO] server-c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa: compacting logs: from=1 to=2 +2024-07-05T13:36:58.513Z [INFO] server-c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa: Installed remote snapshot +2024-07-05T13:36:58.513Z [INFO] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: pipelining replication: peer="{Voter server-33a6508d-5612-e9c8-740b-dd7cb40e9784 33a6508d-5612-e9c8-740b-dd7cb40e9784}" +2024-07-05T13:36:58.513Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:36:58.514Z [INFO] server-07db59d4-a90b-60b3-80aa-89a59e6b3d5f: pipelining replication: peer="{Voter server-c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa c58e16ba-cc53-0f07-fbe6-5b44dcaf7ffa}" +2024-07-05T13:36:59.519Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:36:58.513917666 +0000 UTC m=+84.509838534 (2 nodes), 5.542µs from start of poll, 1.613672948s from cluster start. Timeout at 2024-07-05 13:36:59.51896427 +0000 UTC m=+85.514885138, 1.005046604s after stability +2024-07-05T13:36:59.543Z [INFO] server-071be13a-7f98-f7d3-2a5d-0f657c8dbeac: aborting pipeline replication: peer="{Voter server-285b0f21-70a5-08b0-6dc1-aa822e4cfb45 285b0f21-70a5-08b0-6dc1-aa822e4cfb45}" +2024-07-05T13:36:59.544Z [INFO] server-071be13a-7f98-f7d3-2a5d-0f657c8dbeac: aborting pipeline replication: peer="{Voter server-9b15fc1c-eb9c-02cc-239e-3a802ff2d30f 9b15fc1c-eb9c-02cc-239e-3a802ff2d30f}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false -2024-07-05T13:30:42.221Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15f24eba-5e89-4119-699f-b5a7765c3f41 Address:15f24eba-5e89-4119-699f-b5a7765c3f41} {Suffrage:Voter ID:server-d311a86a-41ee-890c-4a85-4505fe9f58bf Address:d311a86a-41ee-890c-4a85-4505fe9f58bf} {Suffrage:Voter ID:server-4aa05d1b-900b-77a6-fc4d-2c7133d1e18f Address:4aa05d1b-900b-77a6-fc4d-2c7133d1e18f}]" -2024-07-05T13:30:42.221Z [INFO] server-d311a86a-41ee-890c-4a85-4505fe9f58bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15f24eba-5e89-4119-699f-b5a7765c3f41 Address:15f24eba-5e89-4119-699f-b5a7765c3f41} {Suffrage:Voter ID:server-d311a86a-41ee-890c-4a85-4505fe9f58bf Address:d311a86a-41ee-890c-4a85-4505fe9f58bf} {Suffrage:Voter ID:server-4aa05d1b-900b-77a6-fc4d-2c7133d1e18f Address:4aa05d1b-900b-77a6-fc4d-2c7133d1e18f}]" -2024-07-05T13:30:42.222Z [INFO] server-4aa05d1b-900b-77a6-fc4d-2c7133d1e18f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15f24eba-5e89-4119-699f-b5a7765c3f41 Address:15f24eba-5e89-4119-699f-b5a7765c3f41} {Suffrage:Voter ID:server-d311a86a-41ee-890c-4a85-4505fe9f58bf Address:d311a86a-41ee-890c-4a85-4505fe9f58bf} {Suffrage:Voter ID:server-4aa05d1b-900b-77a6-fc4d-2c7133d1e18f Address:4aa05d1b-900b-77a6-fc4d-2c7133d1e18f}]" -2024-07-05T13:30:42.222Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:42.222Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: entering follower state: follower="Node at 15f24eba-5e89-4119-699f-b5a7765c3f41 [Follower]" leader-address= leader-id= -2024-07-05T13:30:42.222Z [INFO] server-d311a86a-41ee-890c-4a85-4505fe9f58bf: entering follower state: follower="Node at d311a86a-41ee-890c-4a85-4505fe9f58bf [Follower]" leader-address= leader-id= -2024-07-05T13:30:42.222Z [INFO] server-4aa05d1b-900b-77a6-fc4d-2c7133d1e18f: entering follower state: follower="Node at 4aa05d1b-900b-77a6-fc4d-2c7133d1e18f [Follower]" leader-address= leader-id= -2024-07-05T13:30:42.733Z [WARN] server-15f24eba-5e89-4119-699f-b5a7765c3f41: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:42.733Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: entering candidate state: node="Node at 15f24eba-5e89-4119-699f-b5a7765c3f41 [Candidate]" term=2 -2024-07-05T13:30:42.733Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: election won: term=2 tally=2 -2024-07-05T13:30:42.733Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: entering leader state: leader="Node at 15f24eba-5e89-4119-699f-b5a7765c3f41 [Leader]" -2024-07-05T13:30:42.733Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: added peer, starting replication: peer=server-d311a86a-41ee-890c-4a85-4505fe9f58bf -2024-07-05T13:30:42.734Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: added peer, starting replication: peer=server-4aa05d1b-900b-77a6-fc4d-2c7133d1e18f -2024-07-05T13:30:42.734Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: pipelining replication: peer="{Voter server-d311a86a-41ee-890c-4a85-4505fe9f58bf d311a86a-41ee-890c-4a85-4505fe9f58bf}" -2024-07-05T13:30:42.734Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: pipelining replication: peer="{Voter server-4aa05d1b-900b-77a6-fc4d-2c7133d1e18f 4aa05d1b-900b-77a6-fc4d-2c7133d1e18f}" -2024-07-05T13:30:43.744Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:42.734589182 +0000 UTC m=+83.504499857 (1 nodes), 512.389624ms from start of poll, 513.102319ms from cluster start. Timeout at 2024-07-05 13:30:43.74451435 +0000 UTC m=+84.514425065, 1.009925208s after stability -2024-07-05T13:30:43.745Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: starting snapshot up to: index=12 -2024-07-05T13:30:43.745Z [INFO] creating new snapshot: path=/tmp/raft1144428661/snapshots/2-12-1720186243745.tmp -2024-07-05T13:30:43.746Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: snapshot complete up to: index=12 -2024-07-05T13:30:43.748Z [INFO] creating new snapshot: path=/tmp/raft1144428661/snapshots/2-10013-1720186243748.tmp -2024-07-05T13:30:43.749Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: copied to local snapshot: bytes=71 -2024-07-05T13:30:43.750Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: snapshot restore progress: id=2-10013-1720186243748 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:43.750Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: restored user snapshot: index=10013 -2024-07-05T13:30:43.750Z [ERROR] server-15f24eba-5e89-4119-699f-b5a7765c3f41: failed to get log: index=23 error="log not found" -2024-07-05T13:30:43.750Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: aborting pipeline replication: peer="{Voter server-4aa05d1b-900b-77a6-fc4d-2c7133d1e18f 4aa05d1b-900b-77a6-fc4d-2c7133d1e18f}" -2024-07-05T13:30:43.750Z [ERROR] server-15f24eba-5e89-4119-699f-b5a7765c3f41: failed to get log: index=23 error="log not found" -2024-07-05T13:30:43.750Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: aborting pipeline replication: peer="{Voter server-d311a86a-41ee-890c-4a85-4505fe9f58bf d311a86a-41ee-890c-4a85-4505fe9f58bf}" -2024-07-05T13:30:43.757Z [ERROR] server-15f24eba-5e89-4119-699f-b5a7765c3f41: failed to get log: index=23 error="log not found" -2024-07-05T13:30:43.758Z [INFO] creating new snapshot: path=/tmp/raft1620962236/snapshots/2-10013-1720186243758.tmp -2024-07-05T13:30:43.758Z [INFO] server-4aa05d1b-900b-77a6-fc4d-2c7133d1e18f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:43.759Z [INFO] server-4aa05d1b-900b-77a6-fc4d-2c7133d1e18f: copied to local snapshot: bytes=71 -2024-07-05T13:30:43.759Z [INFO] server-4aa05d1b-900b-77a6-fc4d-2c7133d1e18f: snapshot restore progress: id=2-10013-1720186243758 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:43.759Z [INFO] server-4aa05d1b-900b-77a6-fc4d-2c7133d1e18f: Installed remote snapshot -2024-07-05T13:30:43.759Z [WARN] server-4aa05d1b-900b-77a6-fc4d-2c7133d1e18f: skipping application of old log: index=22 -2024-07-05T13:30:43.759Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: pipelining replication: peer="{Voter server-4aa05d1b-900b-77a6-fc4d-2c7133d1e18f 4aa05d1b-900b-77a6-fc4d-2c7133d1e18f}" -2024-07-05T13:30:43.760Z [ERROR] server-15f24eba-5e89-4119-699f-b5a7765c3f41: failed to get log: index=23 error="log not found" -2024-07-05T13:30:43.761Z [INFO] creating new snapshot: path=/tmp/raft590775598/snapshots/2-10013-1720186243761.tmp -2024-07-05T13:30:43.761Z [INFO] server-d311a86a-41ee-890c-4a85-4505fe9f58bf: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:43.762Z [INFO] server-d311a86a-41ee-890c-4a85-4505fe9f58bf: copied to local snapshot: bytes=71 -2024-07-05T13:30:43.763Z [INFO] server-d311a86a-41ee-890c-4a85-4505fe9f58bf: snapshot restore progress: id=2-10013-1720186243761 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:43.763Z [INFO] server-d311a86a-41ee-890c-4a85-4505fe9f58bf: Installed remote snapshot -2024-07-05T13:30:43.763Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: pipelining replication: peer="{Voter server-d311a86a-41ee-890c-4a85-4505fe9f58bf d311a86a-41ee-890c-4a85-4505fe9f58bf}" -2024-07-05T13:30:43.764Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:44.774Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:43.764676523 +0000 UTC m=+84.534587198 (2 nodes), 2.72µs from start of poll, 1.54318966s from cluster start. Timeout at 2024-07-05 13:30:44.771038376 +0000 UTC m=+85.540949051, 1.006361853s after stability -2024-07-05T13:30:44.786Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: aborting pipeline replication: peer="{Voter server-4aa05d1b-900b-77a6-fc4d-2c7133d1e18f 4aa05d1b-900b-77a6-fc4d-2c7133d1e18f}" -2024-07-05T13:30:44.787Z [INFO] server-15f24eba-5e89-4119-699f-b5a7765c3f41: aborting pipeline replication: peer="{Voter server-d311a86a-41ee-890c-4a85-4505fe9f58bf d311a86a-41ee-890c-4a85-4505fe9f58bf}" +2024-07-05T13:36:59.553Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85eaca16-93b0-020a-ca93-a928d0ebc0a6 Address:85eaca16-93b0-020a-ca93-a928d0ebc0a6} {Suffrage:Voter ID:server-94835ee1-88a5-62d5-c406-f7d08a81782f Address:94835ee1-88a5-62d5-c406-f7d08a81782f} {Suffrage:Voter ID:server-568e4adc-7222-738f-eb8c-f89e083cd1fe Address:568e4adc-7222-738f-eb8c-f89e083cd1fe}]" +2024-07-05T13:36:59.553Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: entering follower state: follower="Node at 85eaca16-93b0-020a-ca93-a928d0ebc0a6 [Follower]" leader-address= leader-id= +2024-07-05T13:36:59.553Z [INFO] server-94835ee1-88a5-62d5-c406-f7d08a81782f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85eaca16-93b0-020a-ca93-a928d0ebc0a6 Address:85eaca16-93b0-020a-ca93-a928d0ebc0a6} {Suffrage:Voter ID:server-94835ee1-88a5-62d5-c406-f7d08a81782f Address:94835ee1-88a5-62d5-c406-f7d08a81782f} {Suffrage:Voter ID:server-568e4adc-7222-738f-eb8c-f89e083cd1fe Address:568e4adc-7222-738f-eb8c-f89e083cd1fe}]" +2024-07-05T13:36:59.553Z [INFO] server-94835ee1-88a5-62d5-c406-f7d08a81782f: entering follower state: follower="Node at 94835ee1-88a5-62d5-c406-f7d08a81782f [Follower]" leader-address= leader-id= +2024-07-05T13:36:59.554Z [INFO] server-568e4adc-7222-738f-eb8c-f89e083cd1fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85eaca16-93b0-020a-ca93-a928d0ebc0a6 Address:85eaca16-93b0-020a-ca93-a928d0ebc0a6} {Suffrage:Voter ID:server-94835ee1-88a5-62d5-c406-f7d08a81782f Address:94835ee1-88a5-62d5-c406-f7d08a81782f} {Suffrage:Voter ID:server-568e4adc-7222-738f-eb8c-f89e083cd1fe Address:568e4adc-7222-738f-eb8c-f89e083cd1fe}]" +2024-07-05T13:36:59.554Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:36:59.554Z [INFO] server-568e4adc-7222-738f-eb8c-f89e083cd1fe: entering follower state: follower="Node at 568e4adc-7222-738f-eb8c-f89e083cd1fe [Follower]" leader-address= leader-id= +2024-07-05T13:37:00.180Z [WARN] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:00.181Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: entering candidate state: node="Node at 85eaca16-93b0-020a-ca93-a928d0ebc0a6 [Candidate]" term=2 +2024-07-05T13:37:00.181Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: election won: term=2 tally=2 +2024-07-05T13:37:00.182Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: entering leader state: leader="Node at 85eaca16-93b0-020a-ca93-a928d0ebc0a6 [Leader]" +2024-07-05T13:37:00.182Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: added peer, starting replication: peer=server-94835ee1-88a5-62d5-c406-f7d08a81782f +2024-07-05T13:37:00.182Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: added peer, starting replication: peer=server-568e4adc-7222-738f-eb8c-f89e083cd1fe +2024-07-05T13:37:00.183Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: pipelining replication: peer="{Voter server-94835ee1-88a5-62d5-c406-f7d08a81782f 94835ee1-88a5-62d5-c406-f7d08a81782f}" +2024-07-05T13:37:00.183Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: pipelining replication: peer="{Voter server-568e4adc-7222-738f-eb8c-f89e083cd1fe 568e4adc-7222-738f-eb8c-f89e083cd1fe}" +2024-07-05T13:37:01.187Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:00.182219366 +0000 UTC m=+86.178140817 (1 nodes), 627.786743ms from start of poll, 629.684584ms from cluster start. Timeout at 2024-07-05 13:37:01.18733801 +0000 UTC m=+87.183259462, 1.005118645s after stability +2024-07-05T13:37:01.193Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: starting snapshot up to: index=12 +2024-07-05T13:37:01.193Z [INFO] creating new snapshot: path=/tmp/raft4058162280/snapshots/2-12-1720186621193.tmp +2024-07-05T13:37:01.197Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: snapshot complete up to: index=12 +2024-07-05T13:37:01.202Z [INFO] creating new snapshot: path=/tmp/raft4058162280/snapshots/2-10013-1720186621202.tmp +2024-07-05T13:37:01.205Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: copied to local snapshot: bytes=71 +2024-07-05T13:37:01.206Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: snapshot restore progress: id=2-10013-1720186621202 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:01.207Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: restored user snapshot: index=10013 +2024-07-05T13:37:01.207Z [ERROR] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: failed to get log: index=23 error="log not found" +2024-07-05T13:37:01.207Z [ERROR] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: failed to get log: index=23 error="log not found" +2024-07-05T13:37:01.207Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: aborting pipeline replication: peer="{Voter server-568e4adc-7222-738f-eb8c-f89e083cd1fe 568e4adc-7222-738f-eb8c-f89e083cd1fe}" +2024-07-05T13:37:01.207Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: aborting pipeline replication: peer="{Voter server-94835ee1-88a5-62d5-c406-f7d08a81782f 94835ee1-88a5-62d5-c406-f7d08a81782f}" +2024-07-05T13:37:01.214Z [ERROR] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: failed to get log: index=23 error="log not found" +2024-07-05T13:37:01.214Z [ERROR] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: failed to get log: index=23 error="log not found" +2024-07-05T13:37:01.216Z [INFO] creating new snapshot: path=/tmp/raft3032530018/snapshots/2-10013-1720186621216.tmp +2024-07-05T13:37:01.216Z [INFO] creating new snapshot: path=/tmp/raft1957791547/snapshots/2-10013-1720186621216.tmp +2024-07-05T13:37:01.217Z [INFO] server-94835ee1-88a5-62d5-c406-f7d08a81782f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:01.217Z [INFO] server-568e4adc-7222-738f-eb8c-f89e083cd1fe: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:01.219Z [INFO] server-94835ee1-88a5-62d5-c406-f7d08a81782f: copied to local snapshot: bytes=71 +2024-07-05T13:37:01.220Z [INFO] server-94835ee1-88a5-62d5-c406-f7d08a81782f: snapshot restore progress: id=2-10013-1720186621216 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:01.220Z [INFO] server-568e4adc-7222-738f-eb8c-f89e083cd1fe: copied to local snapshot: bytes=71 +2024-07-05T13:37:01.220Z [INFO] server-94835ee1-88a5-62d5-c406-f7d08a81782f: Installed remote snapshot +2024-07-05T13:37:01.221Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: pipelining replication: peer="{Voter server-94835ee1-88a5-62d5-c406-f7d08a81782f 94835ee1-88a5-62d5-c406-f7d08a81782f}" +2024-07-05T13:37:01.221Z [INFO] server-568e4adc-7222-738f-eb8c-f89e083cd1fe: snapshot restore progress: id=2-10013-1720186621216 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:01.221Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:01.221Z [INFO] server-568e4adc-7222-738f-eb8c-f89e083cd1fe: Installed remote snapshot +2024-07-05T13:37:01.222Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: pipelining replication: peer="{Voter server-568e4adc-7222-738f-eb8c-f89e083cd1fe 568e4adc-7222-738f-eb8c-f89e083cd1fe}" +2024-07-05T13:37:02.237Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:01.221587218 +0000 UTC m=+87.217508378 (2 nodes), 6.125µs from start of poll, 1.669052145s from cluster start. Timeout at 2024-07-05 13:37:02.234570803 +0000 UTC m=+88.230491671, 1.012983293s after stability +2024-07-05T13:37:02.253Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: aborting pipeline replication: peer="{Voter server-94835ee1-88a5-62d5-c406-f7d08a81782f 94835ee1-88a5-62d5-c406-f7d08a81782f}" +2024-07-05T13:37:02.253Z [INFO] server-85eaca16-93b0-020a-ca93-a928d0ebc0a6: aborting pipeline replication: peer="{Voter server-568e4adc-7222-738f-eb8c-f89e083cd1fe 568e4adc-7222-738f-eb8c-f89e083cd1fe}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false -2024-07-05T13:30:44.790Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47e402ce-866e-a471-e74d-6c2bc903c2eb Address:47e402ce-866e-a471-e74d-6c2bc903c2eb} {Suffrage:Voter ID:server-317f4001-c3e0-3037-9ea6-c28990a4ced7 Address:317f4001-c3e0-3037-9ea6-c28990a4ced7} {Suffrage:Voter ID:server-41bfc555-5a52-5e1d-8bbc-5620b92e250d Address:41bfc555-5a52-5e1d-8bbc-5620b92e250d}]" -2024-07-05T13:30:44.790Z [INFO] server-317f4001-c3e0-3037-9ea6-c28990a4ced7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47e402ce-866e-a471-e74d-6c2bc903c2eb Address:47e402ce-866e-a471-e74d-6c2bc903c2eb} {Suffrage:Voter ID:server-317f4001-c3e0-3037-9ea6-c28990a4ced7 Address:317f4001-c3e0-3037-9ea6-c28990a4ced7} {Suffrage:Voter ID:server-41bfc555-5a52-5e1d-8bbc-5620b92e250d Address:41bfc555-5a52-5e1d-8bbc-5620b92e250d}]" -2024-07-05T13:30:44.790Z [INFO] server-41bfc555-5a52-5e1d-8bbc-5620b92e250d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47e402ce-866e-a471-e74d-6c2bc903c2eb Address:47e402ce-866e-a471-e74d-6c2bc903c2eb} {Suffrage:Voter ID:server-317f4001-c3e0-3037-9ea6-c28990a4ced7 Address:317f4001-c3e0-3037-9ea6-c28990a4ced7} {Suffrage:Voter ID:server-41bfc555-5a52-5e1d-8bbc-5620b92e250d Address:41bfc555-5a52-5e1d-8bbc-5620b92e250d}]" -2024-07-05T13:30:44.790Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:44.791Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: entering follower state: follower="Node at 47e402ce-866e-a471-e74d-6c2bc903c2eb [Follower]" leader-address= leader-id= -2024-07-05T13:30:44.791Z [INFO] server-317f4001-c3e0-3037-9ea6-c28990a4ced7: entering follower state: follower="Node at 317f4001-c3e0-3037-9ea6-c28990a4ced7 [Follower]" leader-address= leader-id= -2024-07-05T13:30:44.791Z [INFO] server-41bfc555-5a52-5e1d-8bbc-5620b92e250d: entering follower state: follower="Node at 41bfc555-5a52-5e1d-8bbc-5620b92e250d [Follower]" leader-address= leader-id= -2024-07-05T13:30:45.493Z [WARN] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:45.493Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: entering candidate state: node="Node at 47e402ce-866e-a471-e74d-6c2bc903c2eb [Candidate]" term=2 -2024-07-05T13:30:45.493Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: election won: term=2 tally=2 -2024-07-05T13:30:45.493Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: entering leader state: leader="Node at 47e402ce-866e-a471-e74d-6c2bc903c2eb [Leader]" -2024-07-05T13:30:45.493Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: added peer, starting replication: peer=server-317f4001-c3e0-3037-9ea6-c28990a4ced7 -2024-07-05T13:30:45.493Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: added peer, starting replication: peer=server-41bfc555-5a52-5e1d-8bbc-5620b92e250d -2024-07-05T13:30:45.494Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: pipelining replication: peer="{Voter server-317f4001-c3e0-3037-9ea6-c28990a4ced7 317f4001-c3e0-3037-9ea6-c28990a4ced7}" -2024-07-05T13:30:45.494Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: pipelining replication: peer="{Voter server-41bfc555-5a52-5e1d-8bbc-5620b92e250d 41bfc555-5a52-5e1d-8bbc-5620b92e250d}" -2024-07-05T13:30:46.500Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:45.494589871 +0000 UTC m=+86.264500546 (1 nodes), 703.586676ms from start of poll, 704.242733ms from cluster start. Timeout at 2024-07-05 13:30:46.499983197 +0000 UTC m=+87.269893872, 1.005393326s after stability -2024-07-05T13:30:46.507Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: starting snapshot up to: index=12 -2024-07-05T13:30:46.507Z [INFO] creating new snapshot: path=/tmp/raft3188089035/snapshots/2-12-1720186246507.tmp -2024-07-05T13:30:46.509Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: snapshot complete up to: index=12 -2024-07-05T13:30:46.510Z [INFO] creating new snapshot: path=/tmp/raft3188089035/snapshots/2-10013-1720186246510.tmp -2024-07-05T13:30:46.511Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: copied to local snapshot: bytes=71 -2024-07-05T13:30:46.513Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: snapshot restore progress: id=2-10013-1720186246510 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:46.513Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: removing all old logs from log store -2024-07-05T13:30:46.513Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: compacting logs: from=1 to=22 -2024-07-05T13:30:46.513Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: restored user snapshot: index=10013 -2024-07-05T13:30:46.513Z [ERROR] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: failed to get log: index=22 error="log not found" -2024-07-05T13:30:46.513Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: aborting pipeline replication: peer="{Voter server-41bfc555-5a52-5e1d-8bbc-5620b92e250d 41bfc555-5a52-5e1d-8bbc-5620b92e250d}" -2024-07-05T13:30:46.513Z [ERROR] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: failed to get log: index=22 error="log not found" -2024-07-05T13:30:46.513Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: aborting pipeline replication: peer="{Voter server-317f4001-c3e0-3037-9ea6-c28990a4ced7 317f4001-c3e0-3037-9ea6-c28990a4ced7}" -2024-07-05T13:30:46.520Z [ERROR] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: failed to get log: index=22 error="log not found" -2024-07-05T13:30:46.521Z [INFO] creating new snapshot: path=/tmp/raft3637847693/snapshots/2-10013-1720186246521.tmp -2024-07-05T13:30:46.521Z [INFO] server-317f4001-c3e0-3037-9ea6-c28990a4ced7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:46.522Z [INFO] server-317f4001-c3e0-3037-9ea6-c28990a4ced7: copied to local snapshot: bytes=71 -2024-07-05T13:30:46.523Z [ERROR] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: failed to get log: index=22 error="log not found" -2024-07-05T13:30:46.524Z [INFO] creating new snapshot: path=/tmp/raft2978205813/snapshots/2-10013-1720186246524.tmp -2024-07-05T13:30:46.525Z [INFO] server-41bfc555-5a52-5e1d-8bbc-5620b92e250d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:46.525Z [INFO] server-41bfc555-5a52-5e1d-8bbc-5620b92e250d: copied to local snapshot: bytes=71 -2024-07-05T13:30:46.527Z [INFO] server-41bfc555-5a52-5e1d-8bbc-5620b92e250d: snapshot restore progress: id=2-10013-1720186246524 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:46.527Z [INFO] server-41bfc555-5a52-5e1d-8bbc-5620b92e250d: removing all old logs from log store -2024-07-05T13:30:46.527Z [INFO] server-41bfc555-5a52-5e1d-8bbc-5620b92e250d: compacting logs: from=1 to=22 -2024-07-05T13:30:46.527Z [INFO] server-41bfc555-5a52-5e1d-8bbc-5620b92e250d: Installed remote snapshot -2024-07-05T13:30:46.527Z [WARN] server-41bfc555-5a52-5e1d-8bbc-5620b92e250d: skipping application of old log: index=22 -2024-07-05T13:30:46.527Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: pipelining replication: peer="{Voter server-41bfc555-5a52-5e1d-8bbc-5620b92e250d 41bfc555-5a52-5e1d-8bbc-5620b92e250d}" -2024-07-05T13:30:46.527Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:46.528Z [INFO] server-317f4001-c3e0-3037-9ea6-c28990a4ced7: snapshot restore progress: id=2-10013-1720186246521 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:46.528Z [INFO] server-317f4001-c3e0-3037-9ea6-c28990a4ced7: removing all old logs from log store -2024-07-05T13:30:46.528Z [INFO] server-317f4001-c3e0-3037-9ea6-c28990a4ced7: compacting logs: from=1 to=22 -2024-07-05T13:30:46.528Z [INFO] server-317f4001-c3e0-3037-9ea6-c28990a4ced7: Installed remote snapshot -2024-07-05T13:30:46.528Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: pipelining replication: peer="{Voter server-317f4001-c3e0-3037-9ea6-c28990a4ced7 317f4001-c3e0-3037-9ea6-c28990a4ced7}" -2024-07-05T13:30:47.532Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:46.527482033 +0000 UTC m=+87.297392708 (2 nodes), 1.48µs from start of poll, 1.737134895s from cluster start. Timeout at 2024-07-05 13:30:47.532488933 +0000 UTC m=+88.302399608, 1.0050069s after stability -2024-07-05T13:30:47.568Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: aborting pipeline replication: peer="{Voter server-41bfc555-5a52-5e1d-8bbc-5620b92e250d 41bfc555-5a52-5e1d-8bbc-5620b92e250d}" -2024-07-05T13:30:47.568Z [INFO] server-47e402ce-866e-a471-e74d-6c2bc903c2eb: aborting pipeline replication: peer="{Voter server-317f4001-c3e0-3037-9ea6-c28990a4ced7 317f4001-c3e0-3037-9ea6-c28990a4ced7}" +2024-07-05T13:37:02.266Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f Address:ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f} {Suffrage:Voter ID:server-1f4316e8-b8b0-071a-bdaa-1905001bca9f Address:1f4316e8-b8b0-071a-bdaa-1905001bca9f} {Suffrage:Voter ID:server-950e4bf5-3819-5581-5770-7683e4b8a766 Address:950e4bf5-3819-5581-5770-7683e4b8a766}]" +2024-07-05T13:37:02.267Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: entering follower state: follower="Node at ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f [Follower]" leader-address= leader-id= +2024-07-05T13:37:02.267Z [INFO] server-1f4316e8-b8b0-071a-bdaa-1905001bca9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f Address:ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f} {Suffrage:Voter ID:server-1f4316e8-b8b0-071a-bdaa-1905001bca9f Address:1f4316e8-b8b0-071a-bdaa-1905001bca9f} {Suffrage:Voter ID:server-950e4bf5-3819-5581-5770-7683e4b8a766 Address:950e4bf5-3819-5581-5770-7683e4b8a766}]" +2024-07-05T13:37:02.268Z [INFO] server-1f4316e8-b8b0-071a-bdaa-1905001bca9f: entering follower state: follower="Node at 1f4316e8-b8b0-071a-bdaa-1905001bca9f [Follower]" leader-address= leader-id= +2024-07-05T13:37:02.268Z [INFO] server-950e4bf5-3819-5581-5770-7683e4b8a766: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f Address:ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f} {Suffrage:Voter ID:server-1f4316e8-b8b0-071a-bdaa-1905001bca9f Address:1f4316e8-b8b0-071a-bdaa-1905001bca9f} {Suffrage:Voter ID:server-950e4bf5-3819-5581-5770-7683e4b8a766 Address:950e4bf5-3819-5581-5770-7683e4b8a766}]" +2024-07-05T13:37:02.268Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:02.268Z [INFO] server-950e4bf5-3819-5581-5770-7683e4b8a766: entering follower state: follower="Node at 950e4bf5-3819-5581-5770-7683e4b8a766 [Follower]" leader-address= leader-id= +2024-07-05T13:37:02.823Z [WARN] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:02.823Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: entering candidate state: node="Node at ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f [Candidate]" term=2 +2024-07-05T13:37:02.824Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: election won: term=2 tally=2 +2024-07-05T13:37:02.824Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: entering leader state: leader="Node at ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f [Leader]" +2024-07-05T13:37:02.824Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: added peer, starting replication: peer=server-1f4316e8-b8b0-071a-bdaa-1905001bca9f +2024-07-05T13:37:02.824Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: added peer, starting replication: peer=server-950e4bf5-3819-5581-5770-7683e4b8a766 +2024-07-05T13:37:02.825Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: pipelining replication: peer="{Voter server-950e4bf5-3819-5581-5770-7683e4b8a766 950e4bf5-3819-5581-5770-7683e4b8a766}" +2024-07-05T13:37:02.825Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: pipelining replication: peer="{Voter server-1f4316e8-b8b0-071a-bdaa-1905001bca9f 1f4316e8-b8b0-071a-bdaa-1905001bca9f}" +2024-07-05T13:37:03.829Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:02.824425239 +0000 UTC m=+88.820346399 (1 nodes), 555.918473ms from start of poll, 558.016976ms from cluster start. Timeout at 2024-07-05 13:37:03.829477677 +0000 UTC m=+89.825399129, 1.00505273s after stability +2024-07-05T13:37:03.835Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: starting snapshot up to: index=12 +2024-07-05T13:37:03.835Z [INFO] creating new snapshot: path=/tmp/raft3643622445/snapshots/2-12-1720186623835.tmp +2024-07-05T13:37:03.844Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: snapshot complete up to: index=12 +2024-07-05T13:37:03.850Z [INFO] creating new snapshot: path=/tmp/raft3643622445/snapshots/2-10013-1720186623849.tmp +2024-07-05T13:37:03.853Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: copied to local snapshot: bytes=71 +2024-07-05T13:37:03.854Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: snapshot restore progress: id=2-10013-1720186623849 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:03.854Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: removing all old logs from log store +2024-07-05T13:37:03.854Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: compacting logs: from=1 to=22 +2024-07-05T13:37:03.854Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: restored user snapshot: index=10013 +2024-07-05T13:37:03.855Z [ERROR] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: failed to get log: index=22 error="log not found" +2024-07-05T13:37:03.855Z [ERROR] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: failed to get log: index=22 error="log not found" +2024-07-05T13:37:03.855Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: aborting pipeline replication: peer="{Voter server-950e4bf5-3819-5581-5770-7683e4b8a766 950e4bf5-3819-5581-5770-7683e4b8a766}" +2024-07-05T13:37:03.855Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: aborting pipeline replication: peer="{Voter server-1f4316e8-b8b0-071a-bdaa-1905001bca9f 1f4316e8-b8b0-071a-bdaa-1905001bca9f}" +2024-07-05T13:37:03.861Z [ERROR] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: failed to get log: index=22 error="log not found" +2024-07-05T13:37:03.862Z [ERROR] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: failed to get log: index=22 error="log not found" +2024-07-05T13:37:03.864Z [INFO] creating new snapshot: path=/tmp/raft2956856900/snapshots/2-10013-1720186623864.tmp +2024-07-05T13:37:03.864Z [INFO] creating new snapshot: path=/tmp/raft288296187/snapshots/2-10013-1720186623864.tmp +2024-07-05T13:37:03.866Z [INFO] server-1f4316e8-b8b0-071a-bdaa-1905001bca9f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:03.866Z [INFO] server-950e4bf5-3819-5581-5770-7683e4b8a766: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:03.879Z [INFO] server-1f4316e8-b8b0-071a-bdaa-1905001bca9f: copied to local snapshot: bytes=71 +2024-07-05T13:37:03.880Z [INFO] server-1f4316e8-b8b0-071a-bdaa-1905001bca9f: snapshot restore progress: id=2-10013-1720186623864 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:03.880Z [INFO] server-1f4316e8-b8b0-071a-bdaa-1905001bca9f: removing all old logs from log store +2024-07-05T13:37:03.881Z [INFO] server-1f4316e8-b8b0-071a-bdaa-1905001bca9f: compacting logs: from=1 to=22 +2024-07-05T13:37:03.881Z [INFO] server-1f4316e8-b8b0-071a-bdaa-1905001bca9f: Installed remote snapshot +2024-07-05T13:37:03.881Z [WARN] server-1f4316e8-b8b0-071a-bdaa-1905001bca9f: skipping application of old log: index=22 +2024-07-05T13:37:03.881Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: pipelining replication: peer="{Voter server-1f4316e8-b8b0-071a-bdaa-1905001bca9f 1f4316e8-b8b0-071a-bdaa-1905001bca9f}" +2024-07-05T13:37:03.882Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:03.882Z [INFO] server-950e4bf5-3819-5581-5770-7683e4b8a766: copied to local snapshot: bytes=71 +2024-07-05T13:37:03.883Z [INFO] server-950e4bf5-3819-5581-5770-7683e4b8a766: snapshot restore progress: id=2-10013-1720186623864 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:03.883Z [INFO] server-950e4bf5-3819-5581-5770-7683e4b8a766: removing all old logs from log store +2024-07-05T13:37:03.883Z [INFO] server-950e4bf5-3819-5581-5770-7683e4b8a766: compacting logs: from=1 to=22 +2024-07-05T13:37:03.883Z [INFO] server-950e4bf5-3819-5581-5770-7683e4b8a766: Installed remote snapshot +2024-07-05T13:37:03.884Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: pipelining replication: peer="{Voter server-950e4bf5-3819-5581-5770-7683e4b8a766 950e4bf5-3819-5581-5770-7683e4b8a766}" +2024-07-05T13:37:04.887Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:03.88231767 +0000 UTC m=+89.878238539 (2 nodes), 5.833µs from start of poll, 1.615909116s from cluster start. Timeout at 2024-07-05 13:37:04.887355525 +0000 UTC m=+90.883276685, 1.005038146s after stability +2024-07-05T13:37:04.903Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: aborting pipeline replication: peer="{Voter server-950e4bf5-3819-5581-5770-7683e4b8a766 950e4bf5-3819-5581-5770-7683e4b8a766}" +2024-07-05T13:37:04.903Z [INFO] server-ce9ef3d3-5739-aca8-9b05-8eb9b49ce79f: aborting pipeline replication: peer="{Voter server-1f4316e8-b8b0-071a-bdaa-1905001bca9f 1f4316e8-b8b0-071a-bdaa-1905001bca9f}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true -2024-07-05T13:30:47.573Z [INFO] server-e2613ae2-5930-4d12-0a53-2868df887d1c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2613ae2-5930-4d12-0a53-2868df887d1c Address:e2613ae2-5930-4d12-0a53-2868df887d1c} {Suffrage:Voter ID:server-a32d9a76-1064-ce7f-fcbd-b6531ce6207e Address:a32d9a76-1064-ce7f-fcbd-b6531ce6207e} {Suffrage:Voter ID:server-a8e4fd4c-816a-f57c-ea19-34061e491c85 Address:a8e4fd4c-816a-f57c-ea19-34061e491c85}]" -2024-07-05T13:30:47.573Z [INFO] server-a32d9a76-1064-ce7f-fcbd-b6531ce6207e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2613ae2-5930-4d12-0a53-2868df887d1c Address:e2613ae2-5930-4d12-0a53-2868df887d1c} {Suffrage:Voter ID:server-a32d9a76-1064-ce7f-fcbd-b6531ce6207e Address:a32d9a76-1064-ce7f-fcbd-b6531ce6207e} {Suffrage:Voter ID:server-a8e4fd4c-816a-f57c-ea19-34061e491c85 Address:a8e4fd4c-816a-f57c-ea19-34061e491c85}]" -2024-07-05T13:30:47.574Z [INFO] server-a8e4fd4c-816a-f57c-ea19-34061e491c85: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2613ae2-5930-4d12-0a53-2868df887d1c Address:e2613ae2-5930-4d12-0a53-2868df887d1c} {Suffrage:Voter ID:server-a32d9a76-1064-ce7f-fcbd-b6531ce6207e Address:a32d9a76-1064-ce7f-fcbd-b6531ce6207e} {Suffrage:Voter ID:server-a8e4fd4c-816a-f57c-ea19-34061e491c85 Address:a8e4fd4c-816a-f57c-ea19-34061e491c85}]" -2024-07-05T13:30:47.574Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:47.574Z [INFO] server-e2613ae2-5930-4d12-0a53-2868df887d1c: entering follower state: follower="Node at e2613ae2-5930-4d12-0a53-2868df887d1c [Follower]" leader-address= leader-id= -2024-07-05T13:30:47.574Z [INFO] server-a32d9a76-1064-ce7f-fcbd-b6531ce6207e: entering follower state: follower="Node at a32d9a76-1064-ce7f-fcbd-b6531ce6207e [Follower]" leader-address= leader-id= -2024-07-05T13:30:47.574Z [INFO] server-a8e4fd4c-816a-f57c-ea19-34061e491c85: entering follower state: follower="Node at a8e4fd4c-816a-f57c-ea19-34061e491c85 [Follower]" leader-address= leader-id= -2024-07-05T13:30:48.179Z [WARN] server-e2613ae2-5930-4d12-0a53-2868df887d1c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:48.179Z [INFO] server-e2613ae2-5930-4d12-0a53-2868df887d1c: entering candidate state: node="Node at e2613ae2-5930-4d12-0a53-2868df887d1c [Candidate]" term=2 -2024-07-05T13:30:48.180Z [INFO] server-e2613ae2-5930-4d12-0a53-2868df887d1c: election won: term=2 tally=2 -2024-07-05T13:30:48.180Z [INFO] server-e2613ae2-5930-4d12-0a53-2868df887d1c: entering leader state: leader="Node at e2613ae2-5930-4d12-0a53-2868df887d1c [Leader]" -2024-07-05T13:30:48.180Z [INFO] server-e2613ae2-5930-4d12-0a53-2868df887d1c: added peer, starting replication: peer=server-a32d9a76-1064-ce7f-fcbd-b6531ce6207e -2024-07-05T13:30:48.180Z [INFO] server-e2613ae2-5930-4d12-0a53-2868df887d1c: added peer, starting replication: peer=server-a8e4fd4c-816a-f57c-ea19-34061e491c85 -2024-07-05T13:30:48.180Z [INFO] server-e2613ae2-5930-4d12-0a53-2868df887d1c: pipelining replication: peer="{Voter server-a8e4fd4c-816a-f57c-ea19-34061e491c85 a8e4fd4c-816a-f57c-ea19-34061e491c85}" -2024-07-05T13:30:48.180Z [INFO] server-e2613ae2-5930-4d12-0a53-2868df887d1c: pipelining replication: peer="{Voter server-a32d9a76-1064-ce7f-fcbd-b6531ce6207e a32d9a76-1064-ce7f-fcbd-b6531ce6207e}" -2024-07-05T13:30:49.186Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:48.180630154 +0000 UTC m=+88.950540829 (1 nodes), 606.523561ms from start of poll, 607.272975ms from cluster start. Timeout at 2024-07-05 13:30:49.185987722 +0000 UTC m=+89.955898397, 1.005357568s after stability -2024-07-05T13:30:49.187Z [INFO] server-e2613ae2-5930-4d12-0a53-2868df887d1c: starting snapshot up to: index=12 -2024-07-05T13:30:49.187Z [INFO] creating new snapshot: path=/tmp/raft2709426116/snapshots/2-12-1720186249187.tmp -2024-07-05T13:30:49.189Z [INFO] server-e2613ae2-5930-4d12-0a53-2868df887d1c: snapshot complete up to: index=12 -2024-07-05T13:30:49.193Z [INFO] server-7b808d14-a2f4-105c-13ab-f05a51a8cd7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b808d14-a2f4-105c-13ab-f05a51a8cd7d Address:7b808d14-a2f4-105c-13ab-f05a51a8cd7d} {Suffrage:Voter ID:server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e Address:6f7fba6f-09ba-be74-036b-fa7d1c78ff9e} {Suffrage:Voter ID:server-1043a5c0-dacf-99e1-2391-c64d6403d92d Address:1043a5c0-dacf-99e1-2391-c64d6403d92d}]" -2024-07-05T13:30:49.193Z [INFO] server-7b808d14-a2f4-105c-13ab-f05a51a8cd7d: entering follower state: follower="Node at 7b808d14-a2f4-105c-13ab-f05a51a8cd7d [Follower]" leader-address= leader-id= -2024-07-05T13:30:49.194Z [INFO] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b808d14-a2f4-105c-13ab-f05a51a8cd7d Address:7b808d14-a2f4-105c-13ab-f05a51a8cd7d} {Suffrage:Voter ID:server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e Address:6f7fba6f-09ba-be74-036b-fa7d1c78ff9e} {Suffrage:Voter ID:server-1043a5c0-dacf-99e1-2391-c64d6403d92d Address:1043a5c0-dacf-99e1-2391-c64d6403d92d}]" -2024-07-05T13:30:49.194Z [INFO] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: entering follower state: follower="Node at 6f7fba6f-09ba-be74-036b-fa7d1c78ff9e [Follower]" leader-address= leader-id= -2024-07-05T13:30:49.194Z [INFO] server-1043a5c0-dacf-99e1-2391-c64d6403d92d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b808d14-a2f4-105c-13ab-f05a51a8cd7d Address:7b808d14-a2f4-105c-13ab-f05a51a8cd7d} {Suffrage:Voter ID:server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e Address:6f7fba6f-09ba-be74-036b-fa7d1c78ff9e} {Suffrage:Voter ID:server-1043a5c0-dacf-99e1-2391-c64d6403d92d Address:1043a5c0-dacf-99e1-2391-c64d6403d92d}]" -2024-07-05T13:30:49.194Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:49.194Z [INFO] server-1043a5c0-dacf-99e1-2391-c64d6403d92d: entering follower state: follower="Node at 1043a5c0-dacf-99e1-2391-c64d6403d92d [Follower]" leader-address= leader-id= -2024-07-05T13:30:49.733Z [WARN] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:49.733Z [INFO] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: entering candidate state: node="Node at 6f7fba6f-09ba-be74-036b-fa7d1c78ff9e [Candidate]" term=2 -2024-07-05T13:30:49.734Z [INFO] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: election won: term=2 tally=2 -2024-07-05T13:30:49.734Z [INFO] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: entering leader state: leader="Node at 6f7fba6f-09ba-be74-036b-fa7d1c78ff9e [Leader]" -2024-07-05T13:30:49.734Z [INFO] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: added peer, starting replication: peer=server-7b808d14-a2f4-105c-13ab-f05a51a8cd7d -2024-07-05T13:30:49.734Z [INFO] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: added peer, starting replication: peer=server-1043a5c0-dacf-99e1-2391-c64d6403d92d -2024-07-05T13:30:49.735Z [INFO] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: pipelining replication: peer="{Voter server-1043a5c0-dacf-99e1-2391-c64d6403d92d 1043a5c0-dacf-99e1-2391-c64d6403d92d}" -2024-07-05T13:30:49.735Z [INFO] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: pipelining replication: peer="{Voter server-7b808d14-a2f4-105c-13ab-f05a51a8cd7d 7b808d14-a2f4-105c-13ab-f05a51a8cd7d}" -2024-07-05T13:30:50.740Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:49.735176734 +0000 UTC m=+90.505087409 (1 nodes), 540.835145ms from start of poll, 541.694315ms from cluster start. Timeout at 2024-07-05 13:30:50.740809052 +0000 UTC m=+91.510719727, 1.005632318s after stability -2024-07-05T13:30:50.741Z [INFO] creating new snapshot: path=/tmp/raft2063575204/snapshots/2-10013-1720186250741.tmp -2024-07-05T13:30:50.744Z [INFO] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: copied to local snapshot: bytes=71 -2024-07-05T13:30:50.745Z [INFO] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: snapshot restore progress: id=2-10013-1720186250741 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:50.745Z [INFO] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: restored user snapshot: index=10013 -2024-07-05T13:30:50.746Z [ERROR] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: failed to get log: index=3 error="log not found" -2024-07-05T13:30:50.746Z [INFO] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: aborting pipeline replication: peer="{Voter server-7b808d14-a2f4-105c-13ab-f05a51a8cd7d 7b808d14-a2f4-105c-13ab-f05a51a8cd7d}" -2024-07-05T13:30:50.746Z [ERROR] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: failed to get log: index=3 error="log not found" -2024-07-05T13:30:50.746Z [INFO] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: aborting pipeline replication: peer="{Voter server-1043a5c0-dacf-99e1-2391-c64d6403d92d 1043a5c0-dacf-99e1-2391-c64d6403d92d}" -2024-07-05T13:30:50.752Z [ERROR] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: failed to get log: index=3 error="log not found" -2024-07-05T13:30:50.753Z [INFO] creating new snapshot: path=/tmp/raft2667800609/snapshots/2-10013-1720186250753.tmp -2024-07-05T13:30:50.753Z [INFO] server-1043a5c0-dacf-99e1-2391-c64d6403d92d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:50.754Z [ERROR] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: failed to get log: index=3 error="log not found" -2024-07-05T13:30:50.756Z [INFO] creating new snapshot: path=/tmp/raft2951516285/snapshots/2-10013-1720186250756.tmp -2024-07-05T13:30:50.757Z [INFO] server-7b808d14-a2f4-105c-13ab-f05a51a8cd7d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:50.758Z [INFO] server-1043a5c0-dacf-99e1-2391-c64d6403d92d: copied to local snapshot: bytes=71 -2024-07-05T13:30:50.758Z [INFO] server-7b808d14-a2f4-105c-13ab-f05a51a8cd7d: copied to local snapshot: bytes=71 -2024-07-05T13:30:50.759Z [INFO] server-1043a5c0-dacf-99e1-2391-c64d6403d92d: snapshot restore progress: id=2-10013-1720186250753 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:50.759Z [INFO] server-1043a5c0-dacf-99e1-2391-c64d6403d92d: Installed remote snapshot -2024-07-05T13:30:50.760Z [INFO] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: pipelining replication: peer="{Voter server-1043a5c0-dacf-99e1-2391-c64d6403d92d 1043a5c0-dacf-99e1-2391-c64d6403d92d}" -2024-07-05T13:30:50.760Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:50.760Z [INFO] server-7b808d14-a2f4-105c-13ab-f05a51a8cd7d: snapshot restore progress: id=2-10013-1720186250756 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:50.760Z [INFO] server-7b808d14-a2f4-105c-13ab-f05a51a8cd7d: Installed remote snapshot -2024-07-05T13:30:50.761Z [INFO] server-6f7fba6f-09ba-be74-036b-fa7d1c78ff9e: pipelining replication: peer="{Voter server-7b808d14-a2f4-105c-13ab-f05a51a8cd7d 7b808d14-a2f4-105c-13ab-f05a51a8cd7d}" -2024-07-05T13:30:51.765Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:50.76026633 +0000 UTC m=+91.530177005 (2 nodes), 1.6µs from start of poll, 1.566783911s from cluster start. Timeout at 2024-07-05 13:30:51.765460703 +0000 UTC m=+92.535371378, 1.005194373s after stability -2024-07-05T13:30:51.778Z [INFO] server-e2613ae2-5930-4d12-0a53-2868df887d1c: aborting pipeline replication: peer="{Voter server-a8e4fd4c-816a-f57c-ea19-34061e491c85 a8e4fd4c-816a-f57c-ea19-34061e491c85}" -2024-07-05T13:30:51.779Z [INFO] server-e2613ae2-5930-4d12-0a53-2868df887d1c: aborting pipeline replication: peer="{Voter server-a32d9a76-1064-ce7f-fcbd-b6531ce6207e a32d9a76-1064-ce7f-fcbd-b6531ce6207e}" +2024-07-05T13:37:04.916Z [INFO] server-914c8537-2fe2-c6ba-b135-a19c3632fa49: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-914c8537-2fe2-c6ba-b135-a19c3632fa49 Address:914c8537-2fe2-c6ba-b135-a19c3632fa49} {Suffrage:Voter ID:server-be15b29e-5e4f-75c7-c61e-b2529d2e105a Address:be15b29e-5e4f-75c7-c61e-b2529d2e105a} {Suffrage:Voter ID:server-08ea8da2-eac8-4b6b-e765-7b62dd2e531a Address:08ea8da2-eac8-4b6b-e765-7b62dd2e531a}]" +2024-07-05T13:37:04.916Z [INFO] server-914c8537-2fe2-c6ba-b135-a19c3632fa49: entering follower state: follower="Node at 914c8537-2fe2-c6ba-b135-a19c3632fa49 [Follower]" leader-address= leader-id= +2024-07-05T13:37:04.917Z [INFO] server-be15b29e-5e4f-75c7-c61e-b2529d2e105a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-914c8537-2fe2-c6ba-b135-a19c3632fa49 Address:914c8537-2fe2-c6ba-b135-a19c3632fa49} {Suffrage:Voter ID:server-be15b29e-5e4f-75c7-c61e-b2529d2e105a Address:be15b29e-5e4f-75c7-c61e-b2529d2e105a} {Suffrage:Voter ID:server-08ea8da2-eac8-4b6b-e765-7b62dd2e531a Address:08ea8da2-eac8-4b6b-e765-7b62dd2e531a}]" +2024-07-05T13:37:04.917Z [INFO] server-be15b29e-5e4f-75c7-c61e-b2529d2e105a: entering follower state: follower="Node at be15b29e-5e4f-75c7-c61e-b2529d2e105a [Follower]" leader-address= leader-id= +2024-07-05T13:37:04.918Z [INFO] server-08ea8da2-eac8-4b6b-e765-7b62dd2e531a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-914c8537-2fe2-c6ba-b135-a19c3632fa49 Address:914c8537-2fe2-c6ba-b135-a19c3632fa49} {Suffrage:Voter ID:server-be15b29e-5e4f-75c7-c61e-b2529d2e105a Address:be15b29e-5e4f-75c7-c61e-b2529d2e105a} {Suffrage:Voter ID:server-08ea8da2-eac8-4b6b-e765-7b62dd2e531a Address:08ea8da2-eac8-4b6b-e765-7b62dd2e531a}]" +2024-07-05T13:37:04.918Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:04.918Z [INFO] server-08ea8da2-eac8-4b6b-e765-7b62dd2e531a: entering follower state: follower="Node at 08ea8da2-eac8-4b6b-e765-7b62dd2e531a [Follower]" leader-address= leader-id= +2024-07-05T13:37:05.419Z [WARN] server-914c8537-2fe2-c6ba-b135-a19c3632fa49: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:05.419Z [INFO] server-914c8537-2fe2-c6ba-b135-a19c3632fa49: entering candidate state: node="Node at 914c8537-2fe2-c6ba-b135-a19c3632fa49 [Candidate]" term=2 +2024-07-05T13:37:05.419Z [INFO] server-914c8537-2fe2-c6ba-b135-a19c3632fa49: election won: term=2 tally=2 +2024-07-05T13:37:05.420Z [INFO] server-914c8537-2fe2-c6ba-b135-a19c3632fa49: entering leader state: leader="Node at 914c8537-2fe2-c6ba-b135-a19c3632fa49 [Leader]" +2024-07-05T13:37:05.420Z [INFO] server-914c8537-2fe2-c6ba-b135-a19c3632fa49: added peer, starting replication: peer=server-be15b29e-5e4f-75c7-c61e-b2529d2e105a +2024-07-05T13:37:05.420Z [INFO] server-914c8537-2fe2-c6ba-b135-a19c3632fa49: added peer, starting replication: peer=server-08ea8da2-eac8-4b6b-e765-7b62dd2e531a +2024-07-05T13:37:05.422Z [INFO] server-914c8537-2fe2-c6ba-b135-a19c3632fa49: pipelining replication: peer="{Voter server-be15b29e-5e4f-75c7-c61e-b2529d2e105a be15b29e-5e4f-75c7-c61e-b2529d2e105a}" +2024-07-05T13:37:05.422Z [INFO] server-914c8537-2fe2-c6ba-b135-a19c3632fa49: pipelining replication: peer="{Voter server-08ea8da2-eac8-4b6b-e765-7b62dd2e531a 08ea8da2-eac8-4b6b-e765-7b62dd2e531a}" +2024-07-05T13:37:06.428Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:05.422434212 +0000 UTC m=+91.418355372 (1 nodes), 503.721593ms from start of poll, 506.667372ms from cluster start. Timeout at 2024-07-05 13:37:06.428086889 +0000 UTC m=+92.424008341, 1.005652969s after stability +2024-07-05T13:37:06.435Z [INFO] server-914c8537-2fe2-c6ba-b135-a19c3632fa49: starting snapshot up to: index=12 +2024-07-05T13:37:06.436Z [INFO] creating new snapshot: path=/tmp/raft2455029746/snapshots/2-12-1720186626436.tmp +2024-07-05T13:37:06.441Z [INFO] server-914c8537-2fe2-c6ba-b135-a19c3632fa49: snapshot complete up to: index=12 +2024-07-05T13:37:06.461Z [INFO] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45b41c38-c292-ee37-7323-6d9c7079fbd0 Address:45b41c38-c292-ee37-7323-6d9c7079fbd0} {Suffrage:Voter ID:server-5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc Address:5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc} {Suffrage:Voter ID:server-98b7496d-aad5-be56-74c9-a21d30e51c47 Address:98b7496d-aad5-be56-74c9-a21d30e51c47}]" +2024-07-05T13:37:06.461Z [INFO] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: entering follower state: follower="Node at 45b41c38-c292-ee37-7323-6d9c7079fbd0 [Follower]" leader-address= leader-id= +2024-07-05T13:37:06.462Z [INFO] server-5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45b41c38-c292-ee37-7323-6d9c7079fbd0 Address:45b41c38-c292-ee37-7323-6d9c7079fbd0} {Suffrage:Voter ID:server-5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc Address:5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc} {Suffrage:Voter ID:server-98b7496d-aad5-be56-74c9-a21d30e51c47 Address:98b7496d-aad5-be56-74c9-a21d30e51c47}]" +2024-07-05T13:37:06.462Z [INFO] server-5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc: entering follower state: follower="Node at 5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc [Follower]" leader-address= leader-id= +2024-07-05T13:37:06.463Z [INFO] server-98b7496d-aad5-be56-74c9-a21d30e51c47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45b41c38-c292-ee37-7323-6d9c7079fbd0 Address:45b41c38-c292-ee37-7323-6d9c7079fbd0} {Suffrage:Voter ID:server-5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc Address:5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc} {Suffrage:Voter ID:server-98b7496d-aad5-be56-74c9-a21d30e51c47 Address:98b7496d-aad5-be56-74c9-a21d30e51c47}]" +2024-07-05T13:37:06.463Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:06.463Z [INFO] server-98b7496d-aad5-be56-74c9-a21d30e51c47: entering follower state: follower="Node at 98b7496d-aad5-be56-74c9-a21d30e51c47 [Follower]" leader-address= leader-id= +2024-07-05T13:37:07.092Z [WARN] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:07.092Z [INFO] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: entering candidate state: node="Node at 45b41c38-c292-ee37-7323-6d9c7079fbd0 [Candidate]" term=2 +2024-07-05T13:37:07.093Z [INFO] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: election won: term=2 tally=2 +2024-07-05T13:37:07.093Z [INFO] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: entering leader state: leader="Node at 45b41c38-c292-ee37-7323-6d9c7079fbd0 [Leader]" +2024-07-05T13:37:07.093Z [INFO] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: added peer, starting replication: peer=server-5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc +2024-07-05T13:37:07.093Z [INFO] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: added peer, starting replication: peer=server-98b7496d-aad5-be56-74c9-a21d30e51c47 +2024-07-05T13:37:07.094Z [INFO] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: pipelining replication: peer="{Voter server-5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc 5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc}" +2024-07-05T13:37:07.094Z [INFO] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: pipelining replication: peer="{Voter server-98b7496d-aad5-be56-74c9-a21d30e51c47 98b7496d-aad5-be56-74c9-a21d30e51c47}" +2024-07-05T13:37:08.099Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:07.093766274 +0000 UTC m=+93.089687726 (1 nodes), 629.895164ms from start of poll, 633.119188ms from cluster start. Timeout at 2024-07-05 13:37:08.098778463 +0000 UTC m=+94.094699915, 1.005012189s after stability +2024-07-05T13:37:08.100Z [INFO] creating new snapshot: path=/tmp/raft3268397460/snapshots/2-10013-1720186628100.tmp +2024-07-05T13:37:08.104Z [INFO] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: copied to local snapshot: bytes=71 +2024-07-05T13:37:08.108Z [INFO] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: snapshot restore progress: id=2-10013-1720186628100 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:08.109Z [INFO] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: restored user snapshot: index=10013 +2024-07-05T13:37:08.109Z [ERROR] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: failed to get log: index=3 error="log not found" +2024-07-05T13:37:08.109Z [ERROR] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: failed to get log: index=3 error="log not found" +2024-07-05T13:37:08.109Z [INFO] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: aborting pipeline replication: peer="{Voter server-5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc 5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc}" +2024-07-05T13:37:08.109Z [INFO] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: aborting pipeline replication: peer="{Voter server-98b7496d-aad5-be56-74c9-a21d30e51c47 98b7496d-aad5-be56-74c9-a21d30e51c47}" +2024-07-05T13:37:08.117Z [ERROR] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: failed to get log: index=3 error="log not found" +2024-07-05T13:37:08.117Z [ERROR] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: failed to get log: index=3 error="log not found" +2024-07-05T13:37:08.119Z [INFO] creating new snapshot: path=/tmp/raft1368954408/snapshots/2-10013-1720186628119.tmp +2024-07-05T13:37:08.119Z [INFO] creating new snapshot: path=/tmp/raft491998090/snapshots/2-10013-1720186628119.tmp +2024-07-05T13:37:08.120Z [INFO] server-98b7496d-aad5-be56-74c9-a21d30e51c47: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:08.120Z [INFO] server-5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:08.123Z [INFO] server-5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc: copied to local snapshot: bytes=71 +2024-07-05T13:37:08.123Z [INFO] server-98b7496d-aad5-be56-74c9-a21d30e51c47: copied to local snapshot: bytes=71 +2024-07-05T13:37:08.128Z [INFO] server-98b7496d-aad5-be56-74c9-a21d30e51c47: snapshot restore progress: id=2-10013-1720186628119 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:08.135Z [INFO] server-5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc: snapshot restore progress: id=2-10013-1720186628119 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:08.139Z [INFO] server-98b7496d-aad5-be56-74c9-a21d30e51c47: Installed remote snapshot +2024-07-05T13:37:08.141Z [INFO] server-5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc: Installed remote snapshot +2024-07-05T13:37:08.142Z [INFO] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: pipelining replication: peer="{Voter server-98b7496d-aad5-be56-74c9-a21d30e51c47 98b7496d-aad5-be56-74c9-a21d30e51c47}" +2024-07-05T13:37:08.143Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:08.143Z [INFO] server-45b41c38-c292-ee37-7323-6d9c7079fbd0: pipelining replication: peer="{Voter server-5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc 5cf2a8d8-4168-ced6-7aa5-7ed3fa3bd4cc}" +2024-07-05T13:37:09.149Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:08.14310632 +0000 UTC m=+94.139027480 (2 nodes), 5.541µs from start of poll, 1.682458942s from cluster start. Timeout at 2024-07-05 13:37:09.14919299 +0000 UTC m=+95.145114150, 1.00608667s after stability +2024-07-05T13:37:09.163Z [INFO] server-914c8537-2fe2-c6ba-b135-a19c3632fa49: aborting pipeline replication: peer="{Voter server-08ea8da2-eac8-4b6b-e765-7b62dd2e531a 08ea8da2-eac8-4b6b-e765-7b62dd2e531a}" +2024-07-05T13:37:09.164Z [INFO] server-914c8537-2fe2-c6ba-b135-a19c3632fa49: aborting pipeline replication: peer="{Voter server-be15b29e-5e4f-75c7-c61e-b2529d2e105a be15b29e-5e4f-75c7-c61e-b2529d2e105a}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true -2024-07-05T13:30:51.782Z [INFO] server-3fa41167-5615-acb2-e6de-89578ac6b388: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3fa41167-5615-acb2-e6de-89578ac6b388 Address:3fa41167-5615-acb2-e6de-89578ac6b388} {Suffrage:Voter ID:server-e9104203-0339-dccd-5c7e-4d54faf83d86 Address:e9104203-0339-dccd-5c7e-4d54faf83d86} {Suffrage:Voter ID:server-14fdf3eb-8354-5ced-200d-bf4d247ff7df Address:14fdf3eb-8354-5ced-200d-bf4d247ff7df}]" -2024-07-05T13:30:51.782Z [INFO] server-e9104203-0339-dccd-5c7e-4d54faf83d86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3fa41167-5615-acb2-e6de-89578ac6b388 Address:3fa41167-5615-acb2-e6de-89578ac6b388} {Suffrage:Voter ID:server-e9104203-0339-dccd-5c7e-4d54faf83d86 Address:e9104203-0339-dccd-5c7e-4d54faf83d86} {Suffrage:Voter ID:server-14fdf3eb-8354-5ced-200d-bf4d247ff7df Address:14fdf3eb-8354-5ced-200d-bf4d247ff7df}]" -2024-07-05T13:30:51.782Z [INFO] server-14fdf3eb-8354-5ced-200d-bf4d247ff7df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3fa41167-5615-acb2-e6de-89578ac6b388 Address:3fa41167-5615-acb2-e6de-89578ac6b388} {Suffrage:Voter ID:server-e9104203-0339-dccd-5c7e-4d54faf83d86 Address:e9104203-0339-dccd-5c7e-4d54faf83d86} {Suffrage:Voter ID:server-14fdf3eb-8354-5ced-200d-bf4d247ff7df Address:14fdf3eb-8354-5ced-200d-bf4d247ff7df}]" -2024-07-05T13:30:51.782Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:51.783Z [INFO] server-e9104203-0339-dccd-5c7e-4d54faf83d86: entering follower state: follower="Node at e9104203-0339-dccd-5c7e-4d54faf83d86 [Follower]" leader-address= leader-id= -2024-07-05T13:30:51.783Z [INFO] server-14fdf3eb-8354-5ced-200d-bf4d247ff7df: entering follower state: follower="Node at 14fdf3eb-8354-5ced-200d-bf4d247ff7df [Follower]" leader-address= leader-id= -2024-07-05T13:30:51.782Z [INFO] server-3fa41167-5615-acb2-e6de-89578ac6b388: entering follower state: follower="Node at 3fa41167-5615-acb2-e6de-89578ac6b388 [Follower]" leader-address= leader-id= -2024-07-05T13:30:52.308Z [WARN] server-3fa41167-5615-acb2-e6de-89578ac6b388: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:52.308Z [INFO] server-3fa41167-5615-acb2-e6de-89578ac6b388: entering candidate state: node="Node at 3fa41167-5615-acb2-e6de-89578ac6b388 [Candidate]" term=2 -2024-07-05T13:30:52.308Z [INFO] server-3fa41167-5615-acb2-e6de-89578ac6b388: election won: term=2 tally=2 -2024-07-05T13:30:52.308Z [INFO] server-3fa41167-5615-acb2-e6de-89578ac6b388: entering leader state: leader="Node at 3fa41167-5615-acb2-e6de-89578ac6b388 [Leader]" -2024-07-05T13:30:52.308Z [INFO] server-3fa41167-5615-acb2-e6de-89578ac6b388: added peer, starting replication: peer=server-e9104203-0339-dccd-5c7e-4d54faf83d86 -2024-07-05T13:30:52.308Z [INFO] server-3fa41167-5615-acb2-e6de-89578ac6b388: added peer, starting replication: peer=server-14fdf3eb-8354-5ced-200d-bf4d247ff7df -2024-07-05T13:30:52.308Z [INFO] server-3fa41167-5615-acb2-e6de-89578ac6b388: pipelining replication: peer="{Voter server-14fdf3eb-8354-5ced-200d-bf4d247ff7df 14fdf3eb-8354-5ced-200d-bf4d247ff7df}" -2024-07-05T13:30:52.309Z [WARN] server-14fdf3eb-8354-5ced-200d-bf4d247ff7df: rejecting vote request since our last term is greater: candidate=3fa41167-5615-acb2-e6de-89578ac6b388 last-term=2 last-candidate-term=1 -2024-07-05T13:30:52.309Z [INFO] server-3fa41167-5615-acb2-e6de-89578ac6b388: pipelining replication: peer="{Voter server-e9104203-0339-dccd-5c7e-4d54faf83d86 e9104203-0339-dccd-5c7e-4d54faf83d86}" -2024-07-05T13:30:53.317Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:52.309423259 +0000 UTC m=+93.079333934 (1 nodes), 526.916394ms from start of poll, 527.550572ms from cluster start. Timeout at 2024-07-05 13:30:53.317450093 +0000 UTC m=+94.087360768, 1.008026834s after stability -2024-07-05T13:30:53.319Z [INFO] server-3fa41167-5615-acb2-e6de-89578ac6b388: starting snapshot up to: index=12 -2024-07-05T13:30:53.319Z [INFO] creating new snapshot: path=/tmp/raft2957931496/snapshots/2-12-1720186253319.tmp -2024-07-05T13:30:53.322Z [INFO] server-3fa41167-5615-acb2-e6de-89578ac6b388: snapshot complete up to: index=12 -2024-07-05T13:30:53.326Z [INFO] server-1159a7ae-bea5-5f4d-2da8-830b03a00201: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1159a7ae-bea5-5f4d-2da8-830b03a00201 Address:1159a7ae-bea5-5f4d-2da8-830b03a00201} {Suffrage:Voter ID:server-2b3f41c5-190c-4b12-4013-23cd7fb46151 Address:2b3f41c5-190c-4b12-4013-23cd7fb46151} {Suffrage:Voter ID:server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d Address:4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d}]" -2024-07-05T13:30:53.326Z [INFO] server-2b3f41c5-190c-4b12-4013-23cd7fb46151: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1159a7ae-bea5-5f4d-2da8-830b03a00201 Address:1159a7ae-bea5-5f4d-2da8-830b03a00201} {Suffrage:Voter ID:server-2b3f41c5-190c-4b12-4013-23cd7fb46151 Address:2b3f41c5-190c-4b12-4013-23cd7fb46151} {Suffrage:Voter ID:server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d Address:4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d}]" -2024-07-05T13:30:53.326Z [INFO] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1159a7ae-bea5-5f4d-2da8-830b03a00201 Address:1159a7ae-bea5-5f4d-2da8-830b03a00201} {Suffrage:Voter ID:server-2b3f41c5-190c-4b12-4013-23cd7fb46151 Address:2b3f41c5-190c-4b12-4013-23cd7fb46151} {Suffrage:Voter ID:server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d Address:4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d}]" -2024-07-05T13:30:53.326Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:53.326Z [INFO] server-1159a7ae-bea5-5f4d-2da8-830b03a00201: entering follower state: follower="Node at 1159a7ae-bea5-5f4d-2da8-830b03a00201 [Follower]" leader-address= leader-id= -2024-07-05T13:30:53.326Z [INFO] server-2b3f41c5-190c-4b12-4013-23cd7fb46151: entering follower state: follower="Node at 2b3f41c5-190c-4b12-4013-23cd7fb46151 [Follower]" leader-address= leader-id= -2024-07-05T13:30:53.326Z [INFO] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: entering follower state: follower="Node at 4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d [Follower]" leader-address= leader-id= -2024-07-05T13:30:53.945Z [WARN] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:53.945Z [INFO] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: entering candidate state: node="Node at 4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d [Candidate]" term=2 -2024-07-05T13:30:53.945Z [INFO] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: election won: term=2 tally=2 -2024-07-05T13:30:53.945Z [INFO] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: entering leader state: leader="Node at 4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d [Leader]" -2024-07-05T13:30:53.945Z [INFO] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: added peer, starting replication: peer=server-1159a7ae-bea5-5f4d-2da8-830b03a00201 -2024-07-05T13:30:53.945Z [INFO] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: added peer, starting replication: peer=server-2b3f41c5-190c-4b12-4013-23cd7fb46151 -2024-07-05T13:30:53.945Z [INFO] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: pipelining replication: peer="{Voter server-2b3f41c5-190c-4b12-4013-23cd7fb46151 2b3f41c5-190c-4b12-4013-23cd7fb46151}" -2024-07-05T13:30:53.946Z [INFO] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: pipelining replication: peer="{Voter server-1159a7ae-bea5-5f4d-2da8-830b03a00201 1159a7ae-bea5-5f4d-2da8-830b03a00201}" -2024-07-05T13:30:54.952Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:53.946452945 +0000 UTC m=+94.716363620 (1 nodes), 619.737737ms from start of poll, 620.438993ms from cluster start. Timeout at 2024-07-05 13:30:54.952303255 +0000 UTC m=+95.722213930, 1.00585031s after stability -2024-07-05T13:30:54.952Z [INFO] creating new snapshot: path=/tmp/raft1097337454/snapshots/2-10013-1720186254952.tmp -2024-07-05T13:30:54.953Z [INFO] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: copied to local snapshot: bytes=71 -2024-07-05T13:30:54.954Z [INFO] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: snapshot restore progress: id=2-10013-1720186254952 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:54.955Z [INFO] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: removing all old logs from log store -2024-07-05T13:30:54.955Z [INFO] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: compacting logs: from=1 to=2 -2024-07-05T13:30:54.955Z [INFO] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: restored user snapshot: index=10013 -2024-07-05T13:30:54.955Z [ERROR] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: failed to get log: index=2 error="log not found" -2024-07-05T13:30:54.955Z [INFO] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: aborting pipeline replication: peer="{Voter server-2b3f41c5-190c-4b12-4013-23cd7fb46151 2b3f41c5-190c-4b12-4013-23cd7fb46151}" -2024-07-05T13:30:54.955Z [ERROR] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: failed to get log: index=2 error="log not found" -2024-07-05T13:30:54.955Z [INFO] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: aborting pipeline replication: peer="{Voter server-1159a7ae-bea5-5f4d-2da8-830b03a00201 1159a7ae-bea5-5f4d-2da8-830b03a00201}" -2024-07-05T13:30:54.961Z [ERROR] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: failed to get log: index=2 error="log not found" -2024-07-05T13:30:54.962Z [INFO] creating new snapshot: path=/tmp/raft1604605656/snapshots/2-10013-1720186254962.tmp -2024-07-05T13:30:54.962Z [INFO] server-2b3f41c5-190c-4b12-4013-23cd7fb46151: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:54.963Z [INFO] server-2b3f41c5-190c-4b12-4013-23cd7fb46151: copied to local snapshot: bytes=71 -2024-07-05T13:30:54.963Z [ERROR] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: failed to get log: index=2 error="log not found" -2024-07-05T13:30:54.966Z [INFO] creating new snapshot: path=/tmp/raft2978806436/snapshots/2-10013-1720186254966.tmp -2024-07-05T13:30:54.966Z [INFO] server-1159a7ae-bea5-5f4d-2da8-830b03a00201: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:54.967Z [INFO] server-1159a7ae-bea5-5f4d-2da8-830b03a00201: copied to local snapshot: bytes=71 -2024-07-05T13:30:54.968Z [INFO] server-2b3f41c5-190c-4b12-4013-23cd7fb46151: snapshot restore progress: id=2-10013-1720186254962 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:54.968Z [INFO] server-2b3f41c5-190c-4b12-4013-23cd7fb46151: removing all old logs from log store -2024-07-05T13:30:54.968Z [INFO] server-2b3f41c5-190c-4b12-4013-23cd7fb46151: compacting logs: from=1 to=2 -2024-07-05T13:30:54.968Z [INFO] server-2b3f41c5-190c-4b12-4013-23cd7fb46151: Installed remote snapshot -2024-07-05T13:30:54.969Z [INFO] server-1159a7ae-bea5-5f4d-2da8-830b03a00201: snapshot restore progress: id=2-10013-1720186254966 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2024-07-05T13:30:54.969Z [INFO] server-1159a7ae-bea5-5f4d-2da8-830b03a00201: removing all old logs from log store -2024-07-05T13:30:54.969Z [INFO] server-1159a7ae-bea5-5f4d-2da8-830b03a00201: compacting logs: from=1 to=2 -2024-07-05T13:30:54.969Z [INFO] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: pipelining replication: peer="{Voter server-2b3f41c5-190c-4b12-4013-23cd7fb46151 2b3f41c5-190c-4b12-4013-23cd7fb46151}" -2024-07-05T13:30:54.969Z [INFO] server-1159a7ae-bea5-5f4d-2da8-830b03a00201: Installed remote snapshot -2024-07-05T13:30:54.969Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:54.969Z [INFO] server-4b44ec93-5021-a1a0-0d3b-5d69b4a7c76d: pipelining replication: peer="{Voter server-1159a7ae-bea5-5f4d-2da8-830b03a00201 1159a7ae-bea5-5f4d-2da8-830b03a00201}" -2024-07-05T13:30:55.975Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:54.969483373 +0000 UTC m=+95.739394048 (2 nodes), 1.76µs from start of poll, 1.643469421s from cluster start. Timeout at 2024-07-05 13:30:55.975000055 +0000 UTC m=+96.744910730, 1.005516682s after stability -2024-07-05T13:30:55.993Z [INFO] server-3fa41167-5615-acb2-e6de-89578ac6b388: aborting pipeline replication: peer="{Voter server-14fdf3eb-8354-5ced-200d-bf4d247ff7df 14fdf3eb-8354-5ced-200d-bf4d247ff7df}" -2024-07-05T13:30:55.993Z [INFO] server-3fa41167-5615-acb2-e6de-89578ac6b388: aborting pipeline replication: peer="{Voter server-e9104203-0339-dccd-5c7e-4d54faf83d86 e9104203-0339-dccd-5c7e-4d54faf83d86}" ---- PASS: TestRaft_UserRestore (83.30s) - --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.65s) - --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.92s) - --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.55s) - --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.21s) - --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.58s) - --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.77s) - --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.11s) - --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.16s) - --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.66s) - --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.61s) - --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.36s) - --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.38s) +2024-07-05T13:37:09.183Z [INFO] server-f477fca6-2e48-38ca-ec9c-ccea8e44ce54: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f477fca6-2e48-38ca-ec9c-ccea8e44ce54 Address:f477fca6-2e48-38ca-ec9c-ccea8e44ce54} {Suffrage:Voter ID:server-2b161ac6-4b87-36f7-9e8d-1514580ad4f5 Address:2b161ac6-4b87-36f7-9e8d-1514580ad4f5} {Suffrage:Voter ID:server-23e95978-cbbe-2f71-2fda-9679e6b90825 Address:23e95978-cbbe-2f71-2fda-9679e6b90825}]" +2024-07-05T13:37:09.184Z [INFO] server-f477fca6-2e48-38ca-ec9c-ccea8e44ce54: entering follower state: follower="Node at f477fca6-2e48-38ca-ec9c-ccea8e44ce54 [Follower]" leader-address= leader-id= +2024-07-05T13:37:09.184Z [INFO] server-2b161ac6-4b87-36f7-9e8d-1514580ad4f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f477fca6-2e48-38ca-ec9c-ccea8e44ce54 Address:f477fca6-2e48-38ca-ec9c-ccea8e44ce54} {Suffrage:Voter ID:server-2b161ac6-4b87-36f7-9e8d-1514580ad4f5 Address:2b161ac6-4b87-36f7-9e8d-1514580ad4f5} {Suffrage:Voter ID:server-23e95978-cbbe-2f71-2fda-9679e6b90825 Address:23e95978-cbbe-2f71-2fda-9679e6b90825}]" +2024-07-05T13:37:09.185Z [INFO] server-2b161ac6-4b87-36f7-9e8d-1514580ad4f5: entering follower state: follower="Node at 2b161ac6-4b87-36f7-9e8d-1514580ad4f5 [Follower]" leader-address= leader-id= +2024-07-05T13:37:09.185Z [INFO] server-23e95978-cbbe-2f71-2fda-9679e6b90825: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f477fca6-2e48-38ca-ec9c-ccea8e44ce54 Address:f477fca6-2e48-38ca-ec9c-ccea8e44ce54} {Suffrage:Voter ID:server-2b161ac6-4b87-36f7-9e8d-1514580ad4f5 Address:2b161ac6-4b87-36f7-9e8d-1514580ad4f5} {Suffrage:Voter ID:server-23e95978-cbbe-2f71-2fda-9679e6b90825 Address:23e95978-cbbe-2f71-2fda-9679e6b90825}]" +2024-07-05T13:37:09.185Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:09.185Z [INFO] server-23e95978-cbbe-2f71-2fda-9679e6b90825: entering follower state: follower="Node at 23e95978-cbbe-2f71-2fda-9679e6b90825 [Follower]" leader-address= leader-id= +2024-07-05T13:37:09.700Z [WARN] server-2b161ac6-4b87-36f7-9e8d-1514580ad4f5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:09.700Z [INFO] server-2b161ac6-4b87-36f7-9e8d-1514580ad4f5: entering candidate state: node="Node at 2b161ac6-4b87-36f7-9e8d-1514580ad4f5 [Candidate]" term=2 +2024-07-05T13:37:09.701Z [INFO] server-2b161ac6-4b87-36f7-9e8d-1514580ad4f5: election won: term=2 tally=2 +2024-07-05T13:37:09.701Z [INFO] server-2b161ac6-4b87-36f7-9e8d-1514580ad4f5: entering leader state: leader="Node at 2b161ac6-4b87-36f7-9e8d-1514580ad4f5 [Leader]" +2024-07-05T13:37:09.701Z [INFO] server-2b161ac6-4b87-36f7-9e8d-1514580ad4f5: added peer, starting replication: peer=server-f477fca6-2e48-38ca-ec9c-ccea8e44ce54 +2024-07-05T13:37:09.701Z [INFO] server-2b161ac6-4b87-36f7-9e8d-1514580ad4f5: added peer, starting replication: peer=server-23e95978-cbbe-2f71-2fda-9679e6b90825 +2024-07-05T13:37:09.703Z [INFO] server-2b161ac6-4b87-36f7-9e8d-1514580ad4f5: pipelining replication: peer="{Voter server-23e95978-cbbe-2f71-2fda-9679e6b90825 23e95978-cbbe-2f71-2fda-9679e6b90825}" +2024-07-05T13:37:09.703Z [INFO] server-2b161ac6-4b87-36f7-9e8d-1514580ad4f5: pipelining replication: peer="{Voter server-f477fca6-2e48-38ca-ec9c-ccea8e44ce54 f477fca6-2e48-38ca-ec9c-ccea8e44ce54}" +2024-07-05T13:37:10.708Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:09.70241272 +0000 UTC m=+95.698334172 (1 nodes), 516.502776ms from start of poll, 519.816341ms from cluster start. Timeout at 2024-07-05 13:37:10.707783361 +0000 UTC m=+96.703704813, 1.005370641s after stability +2024-07-05T13:37:10.715Z [INFO] server-2b161ac6-4b87-36f7-9e8d-1514580ad4f5: starting snapshot up to: index=12 +2024-07-05T13:37:10.715Z [INFO] creating new snapshot: path=/tmp/raft3731283220/snapshots/2-12-1720186630715.tmp +2024-07-05T13:37:10.719Z [INFO] server-2b161ac6-4b87-36f7-9e8d-1514580ad4f5: snapshot complete up to: index=12 +2024-07-05T13:37:10.742Z [INFO] server-2849f140-1688-4453-e266-43a12e85698f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2849f140-1688-4453-e266-43a12e85698f Address:2849f140-1688-4453-e266-43a12e85698f} {Suffrage:Voter ID:server-bfeb2055-793a-3efb-12f3-a4ecdb89e0ac Address:bfeb2055-793a-3efb-12f3-a4ecdb89e0ac} {Suffrage:Voter ID:server-bb30910a-2016-7f0a-b3d0-e119428075fb Address:bb30910a-2016-7f0a-b3d0-e119428075fb}]" +2024-07-05T13:37:10.742Z [INFO] server-2849f140-1688-4453-e266-43a12e85698f: entering follower state: follower="Node at 2849f140-1688-4453-e266-43a12e85698f [Follower]" leader-address= leader-id= +2024-07-05T13:37:10.742Z [INFO] server-bfeb2055-793a-3efb-12f3-a4ecdb89e0ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2849f140-1688-4453-e266-43a12e85698f Address:2849f140-1688-4453-e266-43a12e85698f} {Suffrage:Voter ID:server-bfeb2055-793a-3efb-12f3-a4ecdb89e0ac Address:bfeb2055-793a-3efb-12f3-a4ecdb89e0ac} {Suffrage:Voter ID:server-bb30910a-2016-7f0a-b3d0-e119428075fb Address:bb30910a-2016-7f0a-b3d0-e119428075fb}]" +2024-07-05T13:37:10.743Z [INFO] server-bfeb2055-793a-3efb-12f3-a4ecdb89e0ac: entering follower state: follower="Node at bfeb2055-793a-3efb-12f3-a4ecdb89e0ac [Follower]" leader-address= leader-id= +2024-07-05T13:37:10.743Z [INFO] server-bb30910a-2016-7f0a-b3d0-e119428075fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2849f140-1688-4453-e266-43a12e85698f Address:2849f140-1688-4453-e266-43a12e85698f} {Suffrage:Voter ID:server-bfeb2055-793a-3efb-12f3-a4ecdb89e0ac Address:bfeb2055-793a-3efb-12f3-a4ecdb89e0ac} {Suffrage:Voter ID:server-bb30910a-2016-7f0a-b3d0-e119428075fb Address:bb30910a-2016-7f0a-b3d0-e119428075fb}]" +2024-07-05T13:37:10.743Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:10.743Z [INFO] server-bb30910a-2016-7f0a-b3d0-e119428075fb: entering follower state: follower="Node at bb30910a-2016-7f0a-b3d0-e119428075fb [Follower]" leader-address= leader-id= +2024-07-05T13:37:11.520Z [WARN] server-bb30910a-2016-7f0a-b3d0-e119428075fb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:11.520Z [INFO] server-bb30910a-2016-7f0a-b3d0-e119428075fb: entering candidate state: node="Node at bb30910a-2016-7f0a-b3d0-e119428075fb [Candidate]" term=2 +2024-07-05T13:37:11.521Z [INFO] server-bb30910a-2016-7f0a-b3d0-e119428075fb: election won: term=2 tally=2 +2024-07-05T13:37:11.521Z [INFO] server-bb30910a-2016-7f0a-b3d0-e119428075fb: entering leader state: leader="Node at bb30910a-2016-7f0a-b3d0-e119428075fb [Leader]" +2024-07-05T13:37:11.521Z [INFO] server-bb30910a-2016-7f0a-b3d0-e119428075fb: added peer, starting replication: peer=server-2849f140-1688-4453-e266-43a12e85698f +2024-07-05T13:37:11.521Z [INFO] server-bb30910a-2016-7f0a-b3d0-e119428075fb: added peer, starting replication: peer=server-bfeb2055-793a-3efb-12f3-a4ecdb89e0ac +2024-07-05T13:37:11.521Z [INFO] server-bb30910a-2016-7f0a-b3d0-e119428075fb: pipelining replication: peer="{Voter server-2849f140-1688-4453-e266-43a12e85698f 2849f140-1688-4453-e266-43a12e85698f}" +2024-07-05T13:37:11.522Z [INFO] server-bb30910a-2016-7f0a-b3d0-e119428075fb: pipelining replication: peer="{Voter server-bfeb2055-793a-3efb-12f3-a4ecdb89e0ac bfeb2055-793a-3efb-12f3-a4ecdb89e0ac}" +2024-07-05T13:37:12.526Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:11.521386379 +0000 UTC m=+97.517307247 (1 nodes), 777.778838ms from start of poll, 779.571389ms from cluster start. Timeout at 2024-07-05 13:37:12.526423067 +0000 UTC m=+98.522344227, 1.00503698s after stability +2024-07-05T13:37:12.527Z [INFO] creating new snapshot: path=/tmp/raft3398423988/snapshots/2-10013-1720186632527.tmp +2024-07-05T13:37:12.530Z [INFO] server-bb30910a-2016-7f0a-b3d0-e119428075fb: copied to local snapshot: bytes=71 +2024-07-05T13:37:12.532Z [INFO] server-bb30910a-2016-7f0a-b3d0-e119428075fb: snapshot restore progress: id=2-10013-1720186632527 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:12.532Z [INFO] server-bb30910a-2016-7f0a-b3d0-e119428075fb: removing all old logs from log store +2024-07-05T13:37:12.532Z [INFO] server-bb30910a-2016-7f0a-b3d0-e119428075fb: compacting logs: from=1 to=2 +2024-07-05T13:37:12.532Z [INFO] server-bb30910a-2016-7f0a-b3d0-e119428075fb: restored user snapshot: index=10013 +2024-07-05T13:37:12.532Z [ERROR] server-bb30910a-2016-7f0a-b3d0-e119428075fb: failed to get log: index=2 error="log not found" +2024-07-05T13:37:12.533Z [INFO] server-bb30910a-2016-7f0a-b3d0-e119428075fb: aborting pipeline replication: peer="{Voter server-bfeb2055-793a-3efb-12f3-a4ecdb89e0ac bfeb2055-793a-3efb-12f3-a4ecdb89e0ac}" +2024-07-05T13:37:12.532Z [ERROR] server-bb30910a-2016-7f0a-b3d0-e119428075fb: failed to get log: index=2 error="log not found" +2024-07-05T13:37:12.533Z [INFO] server-bb30910a-2016-7f0a-b3d0-e119428075fb: aborting pipeline replication: peer="{Voter server-2849f140-1688-4453-e266-43a12e85698f 2849f140-1688-4453-e266-43a12e85698f}" +2024-07-05T13:37:12.541Z [ERROR] server-bb30910a-2016-7f0a-b3d0-e119428075fb: failed to get log: index=2 error="log not found" +2024-07-05T13:37:12.542Z [ERROR] server-bb30910a-2016-7f0a-b3d0-e119428075fb: failed to get log: index=2 error="log not found" +2024-07-05T13:37:12.544Z [INFO] creating new snapshot: path=/tmp/raft3227003966/snapshots/2-10013-1720186632544.tmp +2024-07-05T13:37:12.545Z [INFO] creating new snapshot: path=/tmp/raft1140455376/snapshots/2-10013-1720186632545.tmp +2024-07-05T13:37:12.547Z [INFO] server-2849f140-1688-4453-e266-43a12e85698f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:12.547Z [INFO] server-bfeb2055-793a-3efb-12f3-a4ecdb89e0ac: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:12.549Z [INFO] server-2849f140-1688-4453-e266-43a12e85698f: copied to local snapshot: bytes=71 +2024-07-05T13:37:12.550Z [INFO] server-bfeb2055-793a-3efb-12f3-a4ecdb89e0ac: copied to local snapshot: bytes=71 +2024-07-05T13:37:12.550Z [INFO] server-2849f140-1688-4453-e266-43a12e85698f: snapshot restore progress: id=2-10013-1720186632545 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:12.550Z [INFO] server-2849f140-1688-4453-e266-43a12e85698f: removing all old logs from log store +2024-07-05T13:37:12.550Z [INFO] server-2849f140-1688-4453-e266-43a12e85698f: compacting logs: from=1 to=2 +2024-07-05T13:37:12.550Z [INFO] server-2849f140-1688-4453-e266-43a12e85698f: Installed remote snapshot +2024-07-05T13:37:12.551Z [INFO] server-bfeb2055-793a-3efb-12f3-a4ecdb89e0ac: snapshot restore progress: id=2-10013-1720186632544 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2024-07-05T13:37:12.551Z [INFO] server-bb30910a-2016-7f0a-b3d0-e119428075fb: pipelining replication: peer="{Voter server-2849f140-1688-4453-e266-43a12e85698f 2849f140-1688-4453-e266-43a12e85698f}" +2024-07-05T13:37:12.551Z [INFO] server-bfeb2055-793a-3efb-12f3-a4ecdb89e0ac: removing all old logs from log store +2024-07-05T13:37:12.551Z [INFO] server-bfeb2055-793a-3efb-12f3-a4ecdb89e0ac: compacting logs: from=1 to=2 +2024-07-05T13:37:12.551Z [INFO] server-bfeb2055-793a-3efb-12f3-a4ecdb89e0ac: Installed remote snapshot +2024-07-05T13:37:12.551Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:12.552Z [INFO] server-bb30910a-2016-7f0a-b3d0-e119428075fb: pipelining replication: peer="{Voter server-bfeb2055-793a-3efb-12f3-a4ecdb89e0ac bfeb2055-793a-3efb-12f3-a4ecdb89e0ac}" +2024-07-05T13:37:13.557Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:12.551772813 +0000 UTC m=+98.547693681 (2 nodes), 5.542µs from start of poll, 1.809957823s from cluster start. Timeout at 2024-07-05 13:37:13.556838376 +0000 UTC m=+99.552759244, 1.005065563s after stability +2024-07-05T13:37:13.571Z [INFO] server-2b161ac6-4b87-36f7-9e8d-1514580ad4f5: aborting pipeline replication: peer="{Voter server-f477fca6-2e48-38ca-ec9c-ccea8e44ce54 f477fca6-2e48-38ca-ec9c-ccea8e44ce54}" +2024-07-05T13:37:13.571Z [INFO] server-2b161ac6-4b87-36f7-9e8d-1514580ad4f5: aborting pipeline replication: peer="{Voter server-23e95978-cbbe-2f71-2fda-9679e6b90825 23e95978-cbbe-2f71-2fda-9679e6b90825}" +--- PASS: TestRaft_UserRestore (85.36s) + --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.63s) + --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.59s) + --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.63s) + --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.65s) + --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.88s) + --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.72s) + --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.26s) + --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.57s) + --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.92s) + --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.68s) + --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.23s) + --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.54s) --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.65s) - --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.59s) - --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.54s) - --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.16s) - --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.57s) - --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.58s) - --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.30s) - --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.18s) - --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.57s) - --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.78s) - --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.21s) - --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.21s) + --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.66s) + --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.51s) + --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.33s) + --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.73s) + --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.65s) + --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.28s) + --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.21s) + --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.71s) + --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.65s) + --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.27s) + --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.40s) === RUN TestRaft_SendSnapshotFollower -2024-07-05T13:30:55.998Z [INFO] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65516322-f7d9-42e1-260b-7f20e80ebf8f Address:65516322-f7d9-42e1-260b-7f20e80ebf8f} {Suffrage:Voter ID:server-67494874-433d-d616-21f0-d9cd09eeb00b Address:67494874-433d-d616-21f0-d9cd09eeb00b} {Suffrage:Voter ID:server-09f80915-c046-6e26-6577-f4e5951df801 Address:09f80915-c046-6e26-6577-f4e5951df801}]" -2024-07-05T13:30:55.998Z [INFO] server-67494874-433d-d616-21f0-d9cd09eeb00b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65516322-f7d9-42e1-260b-7f20e80ebf8f Address:65516322-f7d9-42e1-260b-7f20e80ebf8f} {Suffrage:Voter ID:server-67494874-433d-d616-21f0-d9cd09eeb00b Address:67494874-433d-d616-21f0-d9cd09eeb00b} {Suffrage:Voter ID:server-09f80915-c046-6e26-6577-f4e5951df801 Address:09f80915-c046-6e26-6577-f4e5951df801}]" -2024-07-05T13:30:55.998Z [INFO] server-09f80915-c046-6e26-6577-f4e5951df801: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65516322-f7d9-42e1-260b-7f20e80ebf8f Address:65516322-f7d9-42e1-260b-7f20e80ebf8f} {Suffrage:Voter ID:server-67494874-433d-d616-21f0-d9cd09eeb00b Address:67494874-433d-d616-21f0-d9cd09eeb00b} {Suffrage:Voter ID:server-09f80915-c046-6e26-6577-f4e5951df801 Address:09f80915-c046-6e26-6577-f4e5951df801}]" -2024-07-05T13:30:55.998Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:55.998Z [INFO] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: entering follower state: follower="Node at 65516322-f7d9-42e1-260b-7f20e80ebf8f [Follower]" leader-address= leader-id= -2024-07-05T13:30:55.998Z [INFO] server-67494874-433d-d616-21f0-d9cd09eeb00b: entering follower state: follower="Node at 67494874-433d-d616-21f0-d9cd09eeb00b [Follower]" leader-address= leader-id= -2024-07-05T13:30:55.998Z [INFO] server-09f80915-c046-6e26-6577-f4e5951df801: entering follower state: follower="Node at 09f80915-c046-6e26-6577-f4e5951df801 [Follower]" leader-address= leader-id= -2024-07-05T13:30:56.069Z [WARN] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:56.069Z [INFO] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: entering candidate state: node="Node at 65516322-f7d9-42e1-260b-7f20e80ebf8f [Candidate]" term=2 -2024-07-05T13:30:56.069Z [INFO] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: election won: term=2 tally=2 -2024-07-05T13:30:56.069Z [INFO] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: entering leader state: leader="Node at 65516322-f7d9-42e1-260b-7f20e80ebf8f [Leader]" -2024-07-05T13:30:56.069Z [INFO] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: added peer, starting replication: peer=server-67494874-433d-d616-21f0-d9cd09eeb00b -2024-07-05T13:30:56.069Z [INFO] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: added peer, starting replication: peer=server-09f80915-c046-6e26-6577-f4e5951df801 -2024-07-05T13:30:56.069Z [INFO] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: pipelining replication: peer="{Voter server-67494874-433d-d616-21f0-d9cd09eeb00b 67494874-433d-d616-21f0-d9cd09eeb00b}" -2024-07-05T13:30:56.069Z [INFO] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: pipelining replication: peer="{Voter server-09f80915-c046-6e26-6577-f4e5951df801 09f80915-c046-6e26-6577-f4e5951df801}" -2024-07-05T13:30:56.175Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:56.069909805 +0000 UTC m=+96.839820480 (2 nodes), 71.180943ms from start of poll, 71.880158ms from cluster start. Timeout at 2024-07-05 13:30:56.175752411 +0000 UTC m=+96.945663086, 105.842606ms after stability -2024-07-05T13:30:56.175Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:56.281Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:56.175997923 +0000 UTC m=+96.945908598 (1 nodes), 1.92µs from start of poll, 177.968276ms from cluster start. Timeout at 2024-07-05 13:30:56.281012719 +0000 UTC m=+97.050923394, 105.014796ms after stability -2024-07-05T13:30:56.281Z [ERROR] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: failed to pipeline appendEntries: peer="{Voter server-67494874-433d-d616-21f0-d9cd09eeb00b 67494874-433d-d616-21f0-d9cd09eeb00b}" error="append pipeline closed" -2024-07-05T13:30:56.281Z [INFO] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: aborting pipeline replication: peer="{Voter server-67494874-433d-d616-21f0-d9cd09eeb00b 67494874-433d-d616-21f0-d9cd09eeb00b}" -2024-07-05T13:30:56.281Z [ERROR] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: failed to appendEntries to: peer="{Voter server-67494874-433d-d616-21f0-d9cd09eeb00b 67494874-433d-d616-21f0-d9cd09eeb00b}" error="failed to connect to peer: 67494874-433d-d616-21f0-d9cd09eeb00b" -2024-07-05T13:30:56.286Z [ERROR] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: failed to heartbeat to: peer=67494874-433d-d616-21f0-d9cd09eeb00b backoff time=10ms error="failed to connect to peer: 67494874-433d-d616-21f0-d9cd09eeb00b" +2024-07-05T13:37:13.580Z [INFO] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7 Address:fad8f4e6-f129-42fd-591b-55b1bc7ad3d7} {Suffrage:Voter ID:server-f64275fa-ef5d-1404-f42f-d788cc75ce1a Address:f64275fa-ef5d-1404-f42f-d788cc75ce1a} {Suffrage:Voter ID:server-c1ce1825-7da3-2700-305e-84abfaeefa3e Address:c1ce1825-7da3-2700-305e-84abfaeefa3e}]" +2024-07-05T13:37:13.580Z [INFO] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: entering follower state: follower="Node at fad8f4e6-f129-42fd-591b-55b1bc7ad3d7 [Follower]" leader-address= leader-id= +2024-07-05T13:37:13.580Z [INFO] server-f64275fa-ef5d-1404-f42f-d788cc75ce1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7 Address:fad8f4e6-f129-42fd-591b-55b1bc7ad3d7} {Suffrage:Voter ID:server-f64275fa-ef5d-1404-f42f-d788cc75ce1a Address:f64275fa-ef5d-1404-f42f-d788cc75ce1a} {Suffrage:Voter ID:server-c1ce1825-7da3-2700-305e-84abfaeefa3e Address:c1ce1825-7da3-2700-305e-84abfaeefa3e}]" +2024-07-05T13:37:13.581Z [INFO] server-f64275fa-ef5d-1404-f42f-d788cc75ce1a: entering follower state: follower="Node at f64275fa-ef5d-1404-f42f-d788cc75ce1a [Follower]" leader-address= leader-id= +2024-07-05T13:37:13.581Z [INFO] server-c1ce1825-7da3-2700-305e-84abfaeefa3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7 Address:fad8f4e6-f129-42fd-591b-55b1bc7ad3d7} {Suffrage:Voter ID:server-f64275fa-ef5d-1404-f42f-d788cc75ce1a Address:f64275fa-ef5d-1404-f42f-d788cc75ce1a} {Suffrage:Voter ID:server-c1ce1825-7da3-2700-305e-84abfaeefa3e Address:c1ce1825-7da3-2700-305e-84abfaeefa3e}]" +2024-07-05T13:37:13.581Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:13.581Z [INFO] server-c1ce1825-7da3-2700-305e-84abfaeefa3e: entering follower state: follower="Node at c1ce1825-7da3-2700-305e-84abfaeefa3e [Follower]" leader-address= leader-id= +2024-07-05T13:37:13.676Z [WARN] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:13.676Z [INFO] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: entering candidate state: node="Node at fad8f4e6-f129-42fd-591b-55b1bc7ad3d7 [Candidate]" term=2 +2024-07-05T13:37:13.676Z [INFO] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: election won: term=2 tally=2 +2024-07-05T13:37:13.676Z [INFO] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: entering leader state: leader="Node at fad8f4e6-f129-42fd-591b-55b1bc7ad3d7 [Leader]" +2024-07-05T13:37:13.677Z [INFO] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: added peer, starting replication: peer=server-f64275fa-ef5d-1404-f42f-d788cc75ce1a +2024-07-05T13:37:13.677Z [INFO] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: added peer, starting replication: peer=server-c1ce1825-7da3-2700-305e-84abfaeefa3e +2024-07-05T13:37:13.677Z [INFO] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: pipelining replication: peer="{Voter server-c1ce1825-7da3-2700-305e-84abfaeefa3e c1ce1825-7da3-2700-305e-84abfaeefa3e}" +2024-07-05T13:37:13.678Z [INFO] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: pipelining replication: peer="{Voter server-f64275fa-ef5d-1404-f42f-d788cc75ce1a f64275fa-ef5d-1404-f42f-d788cc75ce1a}" +2024-07-05T13:37:13.782Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:13.677637212 +0000 UTC m=+99.673558372 (2 nodes), 95.870833ms from start of poll, 97.925004ms from cluster start. Timeout at 2024-07-05 13:37:13.782714043 +0000 UTC m=+99.778635203, 105.076831ms after stability +2024-07-05T13:37:13.783Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:13.888Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:13.783088536 +0000 UTC m=+99.779009404 (1 nodes), 4.666µs from start of poll, 203.376036ms from cluster start. Timeout at 2024-07-05 13:37:13.888253157 +0000 UTC m=+99.884174025, 105.164621ms after stability +2024-07-05T13:37:13.889Z [ERROR] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: failed to pipeline appendEntries: peer="{Voter server-f64275fa-ef5d-1404-f42f-d788cc75ce1a f64275fa-ef5d-1404-f42f-d788cc75ce1a}" error="append pipeline closed" +2024-07-05T13:37:13.889Z [INFO] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: aborting pipeline replication: peer="{Voter server-f64275fa-ef5d-1404-f42f-d788cc75ce1a f64275fa-ef5d-1404-f42f-d788cc75ce1a}" +2024-07-05T13:37:13.889Z [ERROR] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: failed to appendEntries to: peer="{Voter server-f64275fa-ef5d-1404-f42f-d788cc75ce1a f64275fa-ef5d-1404-f42f-d788cc75ce1a}" error="failed to connect to peer: f64275fa-ef5d-1404-f42f-d788cc75ce1a" +2024-07-05T13:37:13.895Z [ERROR] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: failed to heartbeat to: peer=f64275fa-ef5d-1404-f42f-d788cc75ce1a backoff time=10ms error="failed to connect to peer: f64275fa-ef5d-1404-f42f-d788cc75ce1a" +2024-07-05T13:37:13.900Z [ERROR] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: failed to appendEntries to: peer="{Voter server-f64275fa-ef5d-1404-f42f-d788cc75ce1a f64275fa-ef5d-1404-f42f-d788cc75ce1a}" error="failed to connect to peer: f64275fa-ef5d-1404-f42f-d788cc75ce1a" +2024-07-05T13:37:13.911Z [ERROR] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: failed to appendEntries to: peer="{Voter server-f64275fa-ef5d-1404-f42f-d788cc75ce1a f64275fa-ef5d-1404-f42f-d788cc75ce1a}" error="failed to connect to peer: f64275fa-ef5d-1404-f42f-d788cc75ce1a" +2024-07-05T13:37:13.912Z [ERROR] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: failed to heartbeat to: peer=f64275fa-ef5d-1404-f42f-d788cc75ce1a backoff time=10ms error="failed to connect to peer: f64275fa-ef5d-1404-f42f-d788cc75ce1a" raft_test.go:1627: [INFO] Finished apply without behind follower -2024-07-05T13:30:56.300Z [INFO] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: starting snapshot up to: index=102 -2024-07-05T13:30:56.300Z [INFO] creating new snapshot: path=/tmp/raft597748477/snapshots/2-102-1720186256300.tmp -2024-07-05T13:30:56.301Z [INFO] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: compacting logs: from=1 to=92 -2024-07-05T13:30:56.301Z [INFO] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: snapshot complete up to: index=102 -2024-07-05T13:30:56.302Z [ERROR] server-67494874-433d-d616-21f0-d9cd09eeb00b: failed to take snapshot: error="nothing new to snapshot" -2024-07-05T13:30:56.302Z [INFO] server-09f80915-c046-6e26-6577-f4e5951df801: starting snapshot up to: index=102 -2024-07-05T13:30:56.302Z [INFO] creating new snapshot: path=/tmp/raft2087691279/snapshots/2-102-1720186256302.tmp -2024-07-05T13:30:56.303Z [INFO] server-09f80915-c046-6e26-6577-f4e5951df801: compacting logs: from=1 to=92 -2024-07-05T13:30:56.303Z [INFO] server-09f80915-c046-6e26-6577-f4e5951df801: snapshot complete up to: index=102 -2024-07-05T13:30:56.304Z [ERROR] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: failed to get log: index=2 error="log not found" -2024-07-05T13:30:56.305Z [INFO] creating new snapshot: path=/tmp/raft1483160460/snapshots/2-102-1720186256305.tmp -2024-07-05T13:30:56.305Z [INFO] server-67494874-433d-d616-21f0-d9cd09eeb00b: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% -2024-07-05T13:30:56.306Z [INFO] server-67494874-433d-d616-21f0-d9cd09eeb00b: copied to local snapshot: bytes=693 -2024-07-05T13:30:56.306Z [INFO] server-67494874-433d-d616-21f0-d9cd09eeb00b: snapshot restore progress: id=2-102-1720186256305 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% -2024-07-05T13:30:56.306Z [INFO] server-67494874-433d-d616-21f0-d9cd09eeb00b: Installed remote snapshot -2024-07-05T13:30:56.307Z [WARN] server-67494874-433d-d616-21f0-d9cd09eeb00b: skipping application of old log: index=102 -2024-07-05T13:30:56.307Z [INFO] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: pipelining replication: peer="{Voter server-67494874-433d-d616-21f0-d9cd09eeb00b 67494874-433d-d616-21f0-d9cd09eeb00b}" -2024-07-05T13:30:56.309Z [INFO] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: aborting pipeline replication: peer="{Voter server-09f80915-c046-6e26-6577-f4e5951df801 09f80915-c046-6e26-6577-f4e5951df801}" -2024-07-05T13:30:56.309Z [INFO] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: aborting pipeline replication: peer="{Voter server-67494874-433d-d616-21f0-d9cd09eeb00b 67494874-433d-d616-21f0-d9cd09eeb00b}" -2024-07-05T13:30:56.814Z [ERROR] server-65516322-f7d9-42e1-260b-7f20e80ebf8f: failed to heartbeat to: peer=09f80915-c046-6e26-6577-f4e5951df801 backoff time=10ms error="command timed out" ---- PASS: TestRaft_SendSnapshotFollower (0.82s) +2024-07-05T13:37:13.925Z [INFO] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: starting snapshot up to: index=102 +2024-07-05T13:37:13.925Z [INFO] creating new snapshot: path=/tmp/raft1713963129/snapshots/2-102-1720186633925.tmp +2024-07-05T13:37:13.928Z [ERROR] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: failed to heartbeat to: peer=f64275fa-ef5d-1404-f42f-d788cc75ce1a backoff time=10ms error="failed to connect to peer: f64275fa-ef5d-1404-f42f-d788cc75ce1a" +2024-07-05T13:37:13.928Z [INFO] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: compacting logs: from=1 to=92 +2024-07-05T13:37:13.928Z [INFO] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: snapshot complete up to: index=102 +2024-07-05T13:37:13.928Z [ERROR] server-f64275fa-ef5d-1404-f42f-d788cc75ce1a: failed to take snapshot: error="nothing new to snapshot" +2024-07-05T13:37:13.928Z [INFO] server-c1ce1825-7da3-2700-305e-84abfaeefa3e: starting snapshot up to: index=97 +2024-07-05T13:37:13.929Z [INFO] creating new snapshot: path=/tmp/raft407002223/snapshots/2-97-1720186633929.tmp +2024-07-05T13:37:13.930Z [INFO] server-c1ce1825-7da3-2700-305e-84abfaeefa3e: compacting logs: from=1 to=92 +2024-07-05T13:37:13.930Z [INFO] server-c1ce1825-7da3-2700-305e-84abfaeefa3e: snapshot complete up to: index=97 +2024-07-05T13:37:13.931Z [ERROR] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: failed to get log: index=2 error="log not found" +2024-07-05T13:37:13.932Z [INFO] creating new snapshot: path=/tmp/raft3022825573/snapshots/2-102-1720186633932.tmp +2024-07-05T13:37:13.933Z [INFO] server-f64275fa-ef5d-1404-f42f-d788cc75ce1a: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% +2024-07-05T13:37:13.934Z [INFO] server-f64275fa-ef5d-1404-f42f-d788cc75ce1a: copied to local snapshot: bytes=693 +2024-07-05T13:37:13.935Z [INFO] server-f64275fa-ef5d-1404-f42f-d788cc75ce1a: snapshot restore progress: id=2-102-1720186633932 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% +2024-07-05T13:37:13.935Z [INFO] server-f64275fa-ef5d-1404-f42f-d788cc75ce1a: Installed remote snapshot +2024-07-05T13:37:13.935Z [WARN] server-f64275fa-ef5d-1404-f42f-d788cc75ce1a: skipping application of old log: index=102 +2024-07-05T13:37:13.936Z [INFO] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: pipelining replication: peer="{Voter server-f64275fa-ef5d-1404-f42f-d788cc75ce1a f64275fa-ef5d-1404-f42f-d788cc75ce1a}" +2024-07-05T13:37:13.936Z [INFO] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: aborting pipeline replication: peer="{Voter server-c1ce1825-7da3-2700-305e-84abfaeefa3e c1ce1825-7da3-2700-305e-84abfaeefa3e}" +2024-07-05T13:37:13.936Z [INFO] server-fad8f4e6-f129-42fd-591b-55b1bc7ad3d7: aborting pipeline replication: peer="{Voter server-f64275fa-ef5d-1404-f42f-d788cc75ce1a f64275fa-ef5d-1404-f42f-d788cc75ce1a}" +--- PASS: TestRaft_SendSnapshotFollower (0.36s) === RUN TestRaft_SendSnapshotAndLogsFollower -2024-07-05T13:30:56.817Z [INFO] server-b67c4025-f0f8-5c2d-77ab-78e9556dd870: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b67c4025-f0f8-5c2d-77ab-78e9556dd870 Address:b67c4025-f0f8-5c2d-77ab-78e9556dd870} {Suffrage:Voter ID:server-6db17e35-e2f9-8366-8801-8ae79c67a4e4 Address:6db17e35-e2f9-8366-8801-8ae79c67a4e4} {Suffrage:Voter ID:server-bccb2b9a-a7c3-717c-9665-e9879e2b624b Address:bccb2b9a-a7c3-717c-9665-e9879e2b624b}]" -2024-07-05T13:30:56.817Z [INFO] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b67c4025-f0f8-5c2d-77ab-78e9556dd870 Address:b67c4025-f0f8-5c2d-77ab-78e9556dd870} {Suffrage:Voter ID:server-6db17e35-e2f9-8366-8801-8ae79c67a4e4 Address:6db17e35-e2f9-8366-8801-8ae79c67a4e4} {Suffrage:Voter ID:server-bccb2b9a-a7c3-717c-9665-e9879e2b624b Address:bccb2b9a-a7c3-717c-9665-e9879e2b624b}]" -2024-07-05T13:30:56.817Z [INFO] server-bccb2b9a-a7c3-717c-9665-e9879e2b624b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b67c4025-f0f8-5c2d-77ab-78e9556dd870 Address:b67c4025-f0f8-5c2d-77ab-78e9556dd870} {Suffrage:Voter ID:server-6db17e35-e2f9-8366-8801-8ae79c67a4e4 Address:6db17e35-e2f9-8366-8801-8ae79c67a4e4} {Suffrage:Voter ID:server-bccb2b9a-a7c3-717c-9665-e9879e2b624b Address:bccb2b9a-a7c3-717c-9665-e9879e2b624b}]" -2024-07-05T13:30:56.817Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:56.817Z [INFO] server-b67c4025-f0f8-5c2d-77ab-78e9556dd870: entering follower state: follower="Node at b67c4025-f0f8-5c2d-77ab-78e9556dd870 [Follower]" leader-address= leader-id= -2024-07-05T13:30:56.817Z [INFO] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: entering follower state: follower="Node at 6db17e35-e2f9-8366-8801-8ae79c67a4e4 [Follower]" leader-address= leader-id= -2024-07-05T13:30:56.818Z [INFO] server-bccb2b9a-a7c3-717c-9665-e9879e2b624b: entering follower state: follower="Node at bccb2b9a-a7c3-717c-9665-e9879e2b624b [Follower]" leader-address= leader-id= -2024-07-05T13:30:56.883Z [WARN] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:56.883Z [INFO] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: entering candidate state: node="Node at 6db17e35-e2f9-8366-8801-8ae79c67a4e4 [Candidate]" term=2 -2024-07-05T13:30:56.884Z [INFO] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: election won: term=2 tally=2 -2024-07-05T13:30:56.884Z [INFO] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: entering leader state: leader="Node at 6db17e35-e2f9-8366-8801-8ae79c67a4e4 [Leader]" -2024-07-05T13:30:56.884Z [INFO] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: added peer, starting replication: peer=server-b67c4025-f0f8-5c2d-77ab-78e9556dd870 -2024-07-05T13:30:56.884Z [INFO] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: added peer, starting replication: peer=server-bccb2b9a-a7c3-717c-9665-e9879e2b624b -2024-07-05T13:30:56.884Z [INFO] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: pipelining replication: peer="{Voter server-bccb2b9a-a7c3-717c-9665-e9879e2b624b bccb2b9a-a7c3-717c-9665-e9879e2b624b}" -2024-07-05T13:30:56.884Z [INFO] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: pipelining replication: peer="{Voter server-b67c4025-f0f8-5c2d-77ab-78e9556dd870 b67c4025-f0f8-5c2d-77ab-78e9556dd870}" -2024-07-05T13:30:56.990Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:56.885025607 +0000 UTC m=+97.654936282 (2 nodes), 67.290759ms from start of poll, 67.973735ms from cluster start. Timeout at 2024-07-05 13:30:56.990037283 +0000 UTC m=+97.759947958, 105.011676ms after stability -2024-07-05T13:30:56.990Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:57.095Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:56.990205237 +0000 UTC m=+97.760115912 (1 nodes), 2.2µs from start of poll, 173.153365ms from cluster start. Timeout at 2024-07-05 13:30:57.095531182 +0000 UTC m=+97.865441857, 105.325945ms after stability -2024-07-05T13:30:57.096Z [ERROR] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: failed to pipeline appendEntries: peer="{Voter server-b67c4025-f0f8-5c2d-77ab-78e9556dd870 b67c4025-f0f8-5c2d-77ab-78e9556dd870}" error="append pipeline closed" -2024-07-05T13:30:57.096Z [INFO] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: aborting pipeline replication: peer="{Voter server-b67c4025-f0f8-5c2d-77ab-78e9556dd870 b67c4025-f0f8-5c2d-77ab-78e9556dd870}" -2024-07-05T13:30:57.096Z [ERROR] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: failed to appendEntries to: peer="{Voter server-b67c4025-f0f8-5c2d-77ab-78e9556dd870 b67c4025-f0f8-5c2d-77ab-78e9556dd870}" error="failed to connect to peer: b67c4025-f0f8-5c2d-77ab-78e9556dd870" -2024-07-05T13:30:57.099Z [ERROR] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: failed to heartbeat to: peer=b67c4025-f0f8-5c2d-77ab-78e9556dd870 backoff time=10ms error="failed to connect to peer: b67c4025-f0f8-5c2d-77ab-78e9556dd870" -2024-07-05T13:30:57.121Z [ERROR] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: failed to appendEntries to: peer="{Voter server-b67c4025-f0f8-5c2d-77ab-78e9556dd870 b67c4025-f0f8-5c2d-77ab-78e9556dd870}" error="failed to connect to peer: b67c4025-f0f8-5c2d-77ab-78e9556dd870" +2024-07-05T13:37:13.942Z [INFO] server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61 Address:5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61} {Suffrage:Voter ID:server-03e3c9b1-93e0-430a-5bf8-c5ed5c7ea2e8 Address:03e3c9b1-93e0-430a-5bf8-c5ed5c7ea2e8} {Suffrage:Voter ID:server-48fb626c-e418-2ef7-7a66-6dff4af4b881 Address:48fb626c-e418-2ef7-7a66-6dff4af4b881}]" +2024-07-05T13:37:13.942Z [INFO] server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61: entering follower state: follower="Node at 5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61 [Follower]" leader-address= leader-id= +2024-07-05T13:37:13.942Z [INFO] server-03e3c9b1-93e0-430a-5bf8-c5ed5c7ea2e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61 Address:5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61} {Suffrage:Voter ID:server-03e3c9b1-93e0-430a-5bf8-c5ed5c7ea2e8 Address:03e3c9b1-93e0-430a-5bf8-c5ed5c7ea2e8} {Suffrage:Voter ID:server-48fb626c-e418-2ef7-7a66-6dff4af4b881 Address:48fb626c-e418-2ef7-7a66-6dff4af4b881}]" +2024-07-05T13:37:13.942Z [INFO] server-03e3c9b1-93e0-430a-5bf8-c5ed5c7ea2e8: entering follower state: follower="Node at 03e3c9b1-93e0-430a-5bf8-c5ed5c7ea2e8 [Follower]" leader-address= leader-id= +2024-07-05T13:37:13.942Z [INFO] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61 Address:5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61} {Suffrage:Voter ID:server-03e3c9b1-93e0-430a-5bf8-c5ed5c7ea2e8 Address:03e3c9b1-93e0-430a-5bf8-c5ed5c7ea2e8} {Suffrage:Voter ID:server-48fb626c-e418-2ef7-7a66-6dff4af4b881 Address:48fb626c-e418-2ef7-7a66-6dff4af4b881}]" +2024-07-05T13:37:13.942Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:13.943Z [INFO] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: entering follower state: follower="Node at 48fb626c-e418-2ef7-7a66-6dff4af4b881 [Follower]" leader-address= leader-id= +2024-07-05T13:37:14.026Z [WARN] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:14.026Z [INFO] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: entering candidate state: node="Node at 48fb626c-e418-2ef7-7a66-6dff4af4b881 [Candidate]" term=2 +2024-07-05T13:37:14.026Z [INFO] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: election won: term=2 tally=2 +2024-07-05T13:37:14.026Z [INFO] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: entering leader state: leader="Node at 48fb626c-e418-2ef7-7a66-6dff4af4b881 [Leader]" +2024-07-05T13:37:14.026Z [INFO] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: added peer, starting replication: peer=server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61 +2024-07-05T13:37:14.026Z [INFO] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: added peer, starting replication: peer=server-03e3c9b1-93e0-430a-5bf8-c5ed5c7ea2e8 +2024-07-05T13:37:14.027Z [INFO] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: pipelining replication: peer="{Voter server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61 5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61}" +2024-07-05T13:37:14.027Z [INFO] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: pipelining replication: peer="{Voter server-03e3c9b1-93e0-430a-5bf8-c5ed5c7ea2e8 03e3c9b1-93e0-430a-5bf8-c5ed5c7ea2e8}" +2024-07-05T13:37:14.132Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:14.026761086 +0000 UTC m=+100.022681955 (2 nodes), 83.803345ms from start of poll, 84.745995ms from cluster start. Timeout at 2024-07-05 13:37:14.13179271 +0000 UTC m=+100.127714162, 105.032207ms after stability +2024-07-05T13:37:14.132Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:14.237Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:14.132293784 +0000 UTC m=+100.128215236 (1 nodes), 6.709µs from start of poll, 190.279276ms from cluster start. Timeout at 2024-07-05 13:37:14.237337949 +0000 UTC m=+100.233258818, 105.043582ms after stability +2024-07-05T13:37:14.238Z [ERROR] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: failed to pipeline appendEntries: peer="{Voter server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61 5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61}" error="append pipeline closed" +2024-07-05T13:37:14.238Z [INFO] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: aborting pipeline replication: peer="{Voter server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61 5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61}" +2024-07-05T13:37:14.238Z [ERROR] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: failed to appendEntries to: peer="{Voter server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61 5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61}" error="failed to connect to peer: 5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61" +2024-07-05T13:37:14.246Z [ERROR] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: failed to heartbeat to: peer=5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61 backoff time=10ms error="failed to connect to peer: 5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61" +2024-07-05T13:37:14.248Z [ERROR] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: failed to appendEntries to: peer="{Voter server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61 5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61}" error="failed to connect to peer: 5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61" raft_test.go:1669: [INFO] Finished apply without behind follower -2024-07-05T13:30:57.122Z [ERROR] server-b67c4025-f0f8-5c2d-77ab-78e9556dd870: failed to take snapshot: error="nothing new to snapshot" -2024-07-05T13:30:57.122Z [INFO] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: starting snapshot up to: index=102 -2024-07-05T13:30:57.122Z [INFO] creating new snapshot: path=/tmp/raft577847898/snapshots/2-102-1720186257122.tmp -2024-07-05T13:30:57.123Z [INFO] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: compacting logs: from=1 to=92 -2024-07-05T13:30:57.123Z [INFO] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: snapshot complete up to: index=102 -2024-07-05T13:30:57.124Z [INFO] server-bccb2b9a-a7c3-717c-9665-e9879e2b624b: starting snapshot up to: index=70 -2024-07-05T13:30:57.124Z [INFO] creating new snapshot: path=/tmp/raft3146187488/snapshots/2-70-1720186257124.tmp -2024-07-05T13:30:57.125Z [INFO] server-bccb2b9a-a7c3-717c-9665-e9879e2b624b: compacting logs: from=1 to=70 -2024-07-05T13:30:57.125Z [INFO] server-bccb2b9a-a7c3-717c-9665-e9879e2b624b: snapshot complete up to: index=70 -2024-07-05T13:30:57.128Z [ERROR] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: failed to heartbeat to: peer=b67c4025-f0f8-5c2d-77ab-78e9556dd870 backoff time=10ms error="failed to connect to peer: b67c4025-f0f8-5c2d-77ab-78e9556dd870" -2024-07-05T13:30:57.132Z [ERROR] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: failed to get log: index=2 error="log not found" -2024-07-05T13:30:57.132Z [ERROR] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: failed to install snapshot: id=2-102-1720186257122 error="failed to connect to peer: b67c4025-f0f8-5c2d-77ab-78e9556dd870" -2024-07-05T13:30:57.132Z [ERROR] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: failed to send snapshot to: peer="{Voter server-b67c4025-f0f8-5c2d-77ab-78e9556dd870 b67c4025-f0f8-5c2d-77ab-78e9556dd870}" error="failed to connect to peer: b67c4025-f0f8-5c2d-77ab-78e9556dd870" +2024-07-05T13:37:14.259Z [ERROR] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: failed to appendEntries to: peer="{Voter server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61 5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61}" error="failed to connect to peer: 5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61" +2024-07-05T13:37:14.259Z [ERROR] server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61: failed to take snapshot: error="nothing new to snapshot" +2024-07-05T13:37:14.259Z [INFO] server-03e3c9b1-93e0-430a-5bf8-c5ed5c7ea2e8: starting snapshot up to: index=101 +2024-07-05T13:37:14.259Z [INFO] creating new snapshot: path=/tmp/raft4138685663/snapshots/2-101-1720186634259.tmp +2024-07-05T13:37:14.265Z [ERROR] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: failed to heartbeat to: peer=5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61 backoff time=10ms error="failed to connect to peer: 5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61" +2024-07-05T13:37:14.269Z [INFO] server-03e3c9b1-93e0-430a-5bf8-c5ed5c7ea2e8: compacting logs: from=1 to=92 +2024-07-05T13:37:14.269Z [INFO] server-03e3c9b1-93e0-430a-5bf8-c5ed5c7ea2e8: snapshot complete up to: index=101 +2024-07-05T13:37:14.279Z [INFO] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: starting snapshot up to: index=102 +2024-07-05T13:37:14.279Z [INFO] creating new snapshot: path=/tmp/raft4132209012/snapshots/2-102-1720186634279.tmp +2024-07-05T13:37:14.279Z [ERROR] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: failed to appendEntries to: peer="{Voter server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61 5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61}" error="failed to connect to peer: 5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61" +2024-07-05T13:37:14.281Z [INFO] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: compacting logs: from=1 to=92 +2024-07-05T13:37:14.282Z [INFO] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: snapshot complete up to: index=102 +2024-07-05T13:37:14.286Z [ERROR] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: failed to heartbeat to: peer=5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61 backoff time=10ms error="failed to connect to peer: 5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61" +2024-07-05T13:37:14.287Z [WARN] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: failed to contact: server-id=server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61 time=50.11646ms raft_test.go:1690: [INFO] Finished apply without behind follower -2024-07-05T13:30:57.143Z [WARN] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: failed to contact: server-id=server-b67c4025-f0f8-5c2d-77ab-78e9556dd870 time=50.061123ms -2024-07-05T13:30:57.153Z [ERROR] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: failed to get log: index=2 error="log not found" -2024-07-05T13:30:57.154Z [INFO] creating new snapshot: path=/tmp/raft1223003307/snapshots/2-102-1720186257154.tmp -2024-07-05T13:30:57.155Z [INFO] server-b67c4025-f0f8-5c2d-77ab-78e9556dd870: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% -2024-07-05T13:30:57.156Z [INFO] server-b67c4025-f0f8-5c2d-77ab-78e9556dd870: copied to local snapshot: bytes=693 -2024-07-05T13:30:57.157Z [INFO] server-b67c4025-f0f8-5c2d-77ab-78e9556dd870: snapshot restore progress: id=2-102-1720186257154 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% -2024-07-05T13:30:57.157Z [INFO] server-b67c4025-f0f8-5c2d-77ab-78e9556dd870: Installed remote snapshot -2024-07-05T13:30:57.158Z [INFO] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: pipelining replication: peer="{Voter server-b67c4025-f0f8-5c2d-77ab-78e9556dd870 b67c4025-f0f8-5c2d-77ab-78e9556dd870}" -2024-07-05T13:30:57.159Z [INFO] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: aborting pipeline replication: peer="{Voter server-bccb2b9a-a7c3-717c-9665-e9879e2b624b bccb2b9a-a7c3-717c-9665-e9879e2b624b}" -2024-07-05T13:30:57.160Z [INFO] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: aborting pipeline replication: peer="{Voter server-b67c4025-f0f8-5c2d-77ab-78e9556dd870 b67c4025-f0f8-5c2d-77ab-78e9556dd870}" -2024-07-05T13:30:57.661Z [ERROR] server-6db17e35-e2f9-8366-8801-8ae79c67a4e4: failed to heartbeat to: peer=bccb2b9a-a7c3-717c-9665-e9879e2b624b backoff time=10ms error="command timed out" ---- PASS: TestRaft_SendSnapshotAndLogsFollower (0.85s) +2024-07-05T13:37:14.322Z [ERROR] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: failed to get log: index=2 error="log not found" +2024-07-05T13:37:14.323Z [INFO] creating new snapshot: path=/tmp/raft2612997647/snapshots/2-102-1720186634323.tmp +2024-07-05T13:37:14.324Z [INFO] server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% +2024-07-05T13:37:14.325Z [INFO] server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61: copied to local snapshot: bytes=693 +2024-07-05T13:37:14.325Z [INFO] server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61: snapshot restore progress: id=2-102-1720186634323 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% +2024-07-05T13:37:14.325Z [INFO] server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61: Installed remote snapshot +2024-07-05T13:37:14.326Z [INFO] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: pipelining replication: peer="{Voter server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61 5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61}" +2024-07-05T13:37:14.334Z [INFO] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: aborting pipeline replication: peer="{Voter server-03e3c9b1-93e0-430a-5bf8-c5ed5c7ea2e8 03e3c9b1-93e0-430a-5bf8-c5ed5c7ea2e8}" +2024-07-05T13:37:14.334Z [INFO] server-48fb626c-e418-2ef7-7a66-6dff4af4b881: aborting pipeline replication: peer="{Voter server-5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61 5b81fc9a-3dd2-0bba-d3bf-d99ed8151b61}" +--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.40s) === RUN TestRaft_ReJoinFollower -2024-07-05T13:30:57.669Z [INFO] server-82a992a8-5083-6e83-ee34-8de873c6df0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-82a992a8-5083-6e83-ee34-8de873c6df0d Address:82a992a8-5083-6e83-ee34-8de873c6df0d} {Suffrage:Voter ID:server-c635fcb5-5c71-c07f-8c8a-6fb1a59daa01 Address:c635fcb5-5c71-c07f-8c8a-6fb1a59daa01} {Suffrage:Voter ID:server-1dbd75de-9881-500b-7def-0573e7e77dbb Address:1dbd75de-9881-500b-7def-0573e7e77dbb}]" -2024-07-05T13:30:57.669Z [INFO] server-c635fcb5-5c71-c07f-8c8a-6fb1a59daa01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-82a992a8-5083-6e83-ee34-8de873c6df0d Address:82a992a8-5083-6e83-ee34-8de873c6df0d} {Suffrage:Voter ID:server-c635fcb5-5c71-c07f-8c8a-6fb1a59daa01 Address:c635fcb5-5c71-c07f-8c8a-6fb1a59daa01} {Suffrage:Voter ID:server-1dbd75de-9881-500b-7def-0573e7e77dbb Address:1dbd75de-9881-500b-7def-0573e7e77dbb}]" -2024-07-05T13:30:57.669Z [INFO] server-1dbd75de-9881-500b-7def-0573e7e77dbb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-82a992a8-5083-6e83-ee34-8de873c6df0d Address:82a992a8-5083-6e83-ee34-8de873c6df0d} {Suffrage:Voter ID:server-c635fcb5-5c71-c07f-8c8a-6fb1a59daa01 Address:c635fcb5-5c71-c07f-8c8a-6fb1a59daa01} {Suffrage:Voter ID:server-1dbd75de-9881-500b-7def-0573e7e77dbb Address:1dbd75de-9881-500b-7def-0573e7e77dbb}]" -2024-07-05T13:30:57.669Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:57.669Z [INFO] server-82a992a8-5083-6e83-ee34-8de873c6df0d: entering follower state: follower="Node at 82a992a8-5083-6e83-ee34-8de873c6df0d [Follower]" leader-address= leader-id= -2024-07-05T13:30:57.669Z [INFO] server-c635fcb5-5c71-c07f-8c8a-6fb1a59daa01: entering follower state: follower="Node at c635fcb5-5c71-c07f-8c8a-6fb1a59daa01 [Follower]" leader-address= leader-id= -2024-07-05T13:30:57.669Z [INFO] server-1dbd75de-9881-500b-7def-0573e7e77dbb: entering follower state: follower="Node at 1dbd75de-9881-500b-7def-0573e7e77dbb [Follower]" leader-address= leader-id= -2024-07-05T13:30:57.748Z [WARN] server-1dbd75de-9881-500b-7def-0573e7e77dbb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:57.748Z [INFO] server-1dbd75de-9881-500b-7def-0573e7e77dbb: entering candidate state: node="Node at 1dbd75de-9881-500b-7def-0573e7e77dbb [Candidate]" term=2 -2024-07-05T13:30:57.749Z [INFO] server-1dbd75de-9881-500b-7def-0573e7e77dbb: election won: term=2 tally=2 -2024-07-05T13:30:57.749Z [INFO] server-1dbd75de-9881-500b-7def-0573e7e77dbb: entering leader state: leader="Node at 1dbd75de-9881-500b-7def-0573e7e77dbb [Leader]" -2024-07-05T13:30:57.749Z [INFO] server-1dbd75de-9881-500b-7def-0573e7e77dbb: added peer, starting replication: peer=server-82a992a8-5083-6e83-ee34-8de873c6df0d -2024-07-05T13:30:57.749Z [INFO] server-1dbd75de-9881-500b-7def-0573e7e77dbb: added peer, starting replication: peer=server-c635fcb5-5c71-c07f-8c8a-6fb1a59daa01 -2024-07-05T13:30:57.749Z [INFO] server-1dbd75de-9881-500b-7def-0573e7e77dbb: pipelining replication: peer="{Voter server-c635fcb5-5c71-c07f-8c8a-6fb1a59daa01 c635fcb5-5c71-c07f-8c8a-6fb1a59daa01}" -2024-07-05T13:30:57.749Z [INFO] server-1dbd75de-9881-500b-7def-0573e7e77dbb: pipelining replication: peer="{Voter server-82a992a8-5083-6e83-ee34-8de873c6df0d 82a992a8-5083-6e83-ee34-8de873c6df0d}" -2024-07-05T13:30:57.855Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:57.749391601 +0000 UTC m=+98.519302276 (1 nodes), 79.78324ms from start of poll, 80.570613ms from cluster start. Timeout at 2024-07-05 13:30:57.855208769 +0000 UTC m=+98.625119444, 105.817168ms after stability -2024-07-05T13:30:57.860Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:57.966Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:57.861010765 +0000 UTC m=+98.630921440 (2 nodes), 1.84µs from start of poll, 192.189777ms from cluster start. Timeout at 2024-07-05 13:30:57.966072119 +0000 UTC m=+98.735982794, 105.061354ms after stability -2024-07-05T13:30:57.966Z [INFO] server-1dbd75de-9881-500b-7def-0573e7e77dbb: updating configuration: command=RemoveServer server-id=server-82a992a8-5083-6e83-ee34-8de873c6df0d server-addr= servers="[{Suffrage:Voter ID:server-c635fcb5-5c71-c07f-8c8a-6fb1a59daa01 Address:c635fcb5-5c71-c07f-8c8a-6fb1a59daa01} {Suffrage:Voter ID:server-1dbd75de-9881-500b-7def-0573e7e77dbb Address:1dbd75de-9881-500b-7def-0573e7e77dbb}]" -2024-07-05T13:30:57.966Z [INFO] server-1dbd75de-9881-500b-7def-0573e7e77dbb: removed peer, stopping replication: peer=server-82a992a8-5083-6e83-ee34-8de873c6df0d last-index=3 -2024-07-05T13:30:57.966Z [INFO] server-1dbd75de-9881-500b-7def-0573e7e77dbb: aborting pipeline replication: peer="{Voter server-82a992a8-5083-6e83-ee34-8de873c6df0d 82a992a8-5083-6e83-ee34-8de873c6df0d}" -2024-07-05T13:30:58.072Z [INFO] server-1dbd75de-9881-500b-7def-0573e7e77dbb: updating configuration: command=AddVoter server-id=server-82a992a8-5083-6e83-ee34-8de873c6df0d server-addr=82a992a8-5083-6e83-ee34-8de873c6df0d servers="[{Suffrage:Voter ID:server-c635fcb5-5c71-c07f-8c8a-6fb1a59daa01 Address:c635fcb5-5c71-c07f-8c8a-6fb1a59daa01} {Suffrage:Voter ID:server-1dbd75de-9881-500b-7def-0573e7e77dbb Address:1dbd75de-9881-500b-7def-0573e7e77dbb} {Suffrage:Voter ID:server-82a992a8-5083-6e83-ee34-8de873c6df0d Address:82a992a8-5083-6e83-ee34-8de873c6df0d}]" -2024-07-05T13:30:58.072Z [INFO] server-1dbd75de-9881-500b-7def-0573e7e77dbb: added peer, starting replication: peer=server-82a992a8-5083-6e83-ee34-8de873c6df0d -2024-07-05T13:30:58.072Z [WARN] server-82a992a8-5083-6e83-ee34-8de873c6df0d: failed to get previous log: previous-index=4 last-index=3 error="log not found" -2024-07-05T13:30:58.072Z [WARN] server-1dbd75de-9881-500b-7def-0573e7e77dbb: appendEntries rejected, sending older logs: peer="{Voter server-82a992a8-5083-6e83-ee34-8de873c6df0d 82a992a8-5083-6e83-ee34-8de873c6df0d}" next=4 -2024-07-05T13:30:58.072Z [INFO] server-1dbd75de-9881-500b-7def-0573e7e77dbb: pipelining replication: peer="{Voter server-82a992a8-5083-6e83-ee34-8de873c6df0d 82a992a8-5083-6e83-ee34-8de873c6df0d}" -2024-07-05T13:30:58.073Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:58.179Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:58.073290438 +0000 UTC m=+98.843201113 (1 nodes), 840ns from start of poll, 404.46945ms from cluster start. Timeout at 2024-07-05 13:30:58.179776022 +0000 UTC m=+98.949686697, 106.485584ms after stability -2024-07-05T13:30:58.180Z [INFO] server-1dbd75de-9881-500b-7def-0573e7e77dbb: aborting pipeline replication: peer="{Voter server-c635fcb5-5c71-c07f-8c8a-6fb1a59daa01 c635fcb5-5c71-c07f-8c8a-6fb1a59daa01}" -2024-07-05T13:30:58.180Z [INFO] server-1dbd75de-9881-500b-7def-0573e7e77dbb: aborting pipeline replication: peer="{Voter server-82a992a8-5083-6e83-ee34-8de873c6df0d 82a992a8-5083-6e83-ee34-8de873c6df0d}" ---- PASS: TestRaft_ReJoinFollower (0.52s) +2024-07-05T13:37:14.342Z [INFO] server-3977d6a9-eb62-5d15-3522-1787c45237e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3977d6a9-eb62-5d15-3522-1787c45237e7 Address:3977d6a9-eb62-5d15-3522-1787c45237e7} {Suffrage:Voter ID:server-e8f5cf04-eb1e-c1cc-f0ed-122381f1e1ce Address:e8f5cf04-eb1e-c1cc-f0ed-122381f1e1ce} {Suffrage:Voter ID:server-73d27645-7c1f-707a-1854-65976e39d70f Address:73d27645-7c1f-707a-1854-65976e39d70f}]" +2024-07-05T13:37:14.343Z [INFO] server-e8f5cf04-eb1e-c1cc-f0ed-122381f1e1ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3977d6a9-eb62-5d15-3522-1787c45237e7 Address:3977d6a9-eb62-5d15-3522-1787c45237e7} {Suffrage:Voter ID:server-e8f5cf04-eb1e-c1cc-f0ed-122381f1e1ce Address:e8f5cf04-eb1e-c1cc-f0ed-122381f1e1ce} {Suffrage:Voter ID:server-73d27645-7c1f-707a-1854-65976e39d70f Address:73d27645-7c1f-707a-1854-65976e39d70f}]" +2024-07-05T13:37:14.344Z [INFO] server-3977d6a9-eb62-5d15-3522-1787c45237e7: entering follower state: follower="Node at 3977d6a9-eb62-5d15-3522-1787c45237e7 [Follower]" leader-address= leader-id= +2024-07-05T13:37:14.344Z [INFO] server-73d27645-7c1f-707a-1854-65976e39d70f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3977d6a9-eb62-5d15-3522-1787c45237e7 Address:3977d6a9-eb62-5d15-3522-1787c45237e7} {Suffrage:Voter ID:server-e8f5cf04-eb1e-c1cc-f0ed-122381f1e1ce Address:e8f5cf04-eb1e-c1cc-f0ed-122381f1e1ce} {Suffrage:Voter ID:server-73d27645-7c1f-707a-1854-65976e39d70f Address:73d27645-7c1f-707a-1854-65976e39d70f}]" +2024-07-05T13:37:14.344Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:14.346Z [INFO] server-e8f5cf04-eb1e-c1cc-f0ed-122381f1e1ce: entering follower state: follower="Node at e8f5cf04-eb1e-c1cc-f0ed-122381f1e1ce [Follower]" leader-address= leader-id= +2024-07-05T13:37:14.349Z [INFO] server-73d27645-7c1f-707a-1854-65976e39d70f: entering follower state: follower="Node at 73d27645-7c1f-707a-1854-65976e39d70f [Follower]" leader-address= leader-id= +2024-07-05T13:37:14.406Z [WARN] server-73d27645-7c1f-707a-1854-65976e39d70f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:14.406Z [INFO] server-73d27645-7c1f-707a-1854-65976e39d70f: entering candidate state: node="Node at 73d27645-7c1f-707a-1854-65976e39d70f [Candidate]" term=2 +2024-07-05T13:37:14.407Z [INFO] server-73d27645-7c1f-707a-1854-65976e39d70f: election won: term=2 tally=2 +2024-07-05T13:37:14.407Z [INFO] server-73d27645-7c1f-707a-1854-65976e39d70f: entering leader state: leader="Node at 73d27645-7c1f-707a-1854-65976e39d70f [Leader]" +2024-07-05T13:37:14.407Z [INFO] server-73d27645-7c1f-707a-1854-65976e39d70f: added peer, starting replication: peer=server-3977d6a9-eb62-5d15-3522-1787c45237e7 +2024-07-05T13:37:14.407Z [INFO] server-73d27645-7c1f-707a-1854-65976e39d70f: added peer, starting replication: peer=server-e8f5cf04-eb1e-c1cc-f0ed-122381f1e1ce +2024-07-05T13:37:14.408Z [WARN] server-e8f5cf04-eb1e-c1cc-f0ed-122381f1e1ce: rejecting vote request since our last term is greater: candidate=73d27645-7c1f-707a-1854-65976e39d70f last-term=2 last-candidate-term=1 +2024-07-05T13:37:14.408Z [INFO] server-73d27645-7c1f-707a-1854-65976e39d70f: pipelining replication: peer="{Voter server-3977d6a9-eb62-5d15-3522-1787c45237e7 3977d6a9-eb62-5d15-3522-1787c45237e7}" +2024-07-05T13:37:14.408Z [INFO] server-73d27645-7c1f-707a-1854-65976e39d70f: pipelining replication: peer="{Voter server-e8f5cf04-eb1e-c1cc-f0ed-122381f1e1ce e8f5cf04-eb1e-c1cc-f0ed-122381f1e1ce}" +2024-07-05T13:37:14.513Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:14.408497074 +0000 UTC m=+100.404418234 (1 nodes), 64.009456ms from start of poll, 66.643158ms from cluster start. Timeout at 2024-07-05 13:37:14.513766401 +0000 UTC m=+100.509687270, 105.269036ms after stability +2024-07-05T13:37:14.519Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:14.624Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:14.519155136 +0000 UTC m=+100.515076005 (2 nodes), 4.084µs from start of poll, 177.300929ms from cluster start. Timeout at 2024-07-05 13:37:14.624256759 +0000 UTC m=+100.620178794, 105.102789ms after stability +2024-07-05T13:37:14.624Z [INFO] server-73d27645-7c1f-707a-1854-65976e39d70f: updating configuration: command=RemoveServer server-id=server-3977d6a9-eb62-5d15-3522-1787c45237e7 server-addr= servers="[{Suffrage:Voter ID:server-e8f5cf04-eb1e-c1cc-f0ed-122381f1e1ce Address:e8f5cf04-eb1e-c1cc-f0ed-122381f1e1ce} {Suffrage:Voter ID:server-73d27645-7c1f-707a-1854-65976e39d70f Address:73d27645-7c1f-707a-1854-65976e39d70f}]" +2024-07-05T13:37:14.625Z [INFO] server-73d27645-7c1f-707a-1854-65976e39d70f: removed peer, stopping replication: peer=server-3977d6a9-eb62-5d15-3522-1787c45237e7 last-index=3 +2024-07-05T13:37:14.625Z [INFO] server-73d27645-7c1f-707a-1854-65976e39d70f: aborting pipeline replication: peer="{Voter server-3977d6a9-eb62-5d15-3522-1787c45237e7 3977d6a9-eb62-5d15-3522-1787c45237e7}" +2024-07-05T13:37:14.731Z [INFO] server-73d27645-7c1f-707a-1854-65976e39d70f: updating configuration: command=AddVoter server-id=server-3977d6a9-eb62-5d15-3522-1787c45237e7 server-addr=3977d6a9-eb62-5d15-3522-1787c45237e7 servers="[{Suffrage:Voter ID:server-e8f5cf04-eb1e-c1cc-f0ed-122381f1e1ce Address:e8f5cf04-eb1e-c1cc-f0ed-122381f1e1ce} {Suffrage:Voter ID:server-73d27645-7c1f-707a-1854-65976e39d70f Address:73d27645-7c1f-707a-1854-65976e39d70f} {Suffrage:Voter ID:server-3977d6a9-eb62-5d15-3522-1787c45237e7 Address:3977d6a9-eb62-5d15-3522-1787c45237e7}]" +2024-07-05T13:37:14.732Z [INFO] server-73d27645-7c1f-707a-1854-65976e39d70f: added peer, starting replication: peer=server-3977d6a9-eb62-5d15-3522-1787c45237e7 +2024-07-05T13:37:14.732Z [WARN] server-3977d6a9-eb62-5d15-3522-1787c45237e7: failed to get previous log: previous-index=4 last-index=3 error="log not found" +2024-07-05T13:37:14.733Z [WARN] server-73d27645-7c1f-707a-1854-65976e39d70f: appendEntries rejected, sending older logs: peer="{Voter server-3977d6a9-eb62-5d15-3522-1787c45237e7 3977d6a9-eb62-5d15-3522-1787c45237e7}" next=4 +2024-07-05T13:37:14.733Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:14.734Z [INFO] server-73d27645-7c1f-707a-1854-65976e39d70f: pipelining replication: peer="{Voter server-3977d6a9-eb62-5d15-3522-1787c45237e7 3977d6a9-eb62-5d15-3522-1787c45237e7}" +2024-07-05T13:37:14.838Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:14.733715802 +0000 UTC m=+100.729636961 (1 nodes), 4.374µs from start of poll, 391.861885ms from cluster start. Timeout at 2024-07-05 13:37:14.838766092 +0000 UTC m=+100.834687252, 105.050291ms after stability +2024-07-05T13:37:14.839Z [INFO] server-73d27645-7c1f-707a-1854-65976e39d70f: aborting pipeline replication: peer="{Voter server-3977d6a9-eb62-5d15-3522-1787c45237e7 3977d6a9-eb62-5d15-3522-1787c45237e7}" +2024-07-05T13:37:14.839Z [INFO] server-73d27645-7c1f-707a-1854-65976e39d70f: aborting pipeline replication: peer="{Voter server-e8f5cf04-eb1e-c1cc-f0ed-122381f1e1ce e8f5cf04-eb1e-c1cc-f0ed-122381f1e1ce}" +--- PASS: TestRaft_ReJoinFollower (0.51s) === RUN TestRaft_LeaderLeaseExpire -2024-07-05T13:30:58.182Z [INFO] server-45e32979-22cd-bf0d-941c-17bfc220c7f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45e32979-22cd-bf0d-941c-17bfc220c7f1 Address:45e32979-22cd-bf0d-941c-17bfc220c7f1} {Suffrage:Voter ID:server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104 Address:ab2efbb1-f3f9-e20b-5960-a09c2f2f9104}]" -2024-07-05T13:30:58.182Z [INFO] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45e32979-22cd-bf0d-941c-17bfc220c7f1 Address:45e32979-22cd-bf0d-941c-17bfc220c7f1} {Suffrage:Voter ID:server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104 Address:ab2efbb1-f3f9-e20b-5960-a09c2f2f9104}]" -2024-07-05T13:30:58.182Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:58.182Z [INFO] server-45e32979-22cd-bf0d-941c-17bfc220c7f1: entering follower state: follower="Node at 45e32979-22cd-bf0d-941c-17bfc220c7f1 [Follower]" leader-address= leader-id= -2024-07-05T13:30:58.182Z [INFO] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: entering follower state: follower="Node at ab2efbb1-f3f9-e20b-5960-a09c2f2f9104 [Follower]" leader-address= leader-id= -2024-07-05T13:30:58.234Z [WARN] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:58.234Z [INFO] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: entering candidate state: node="Node at ab2efbb1-f3f9-e20b-5960-a09c2f2f9104 [Candidate]" term=2 -2024-07-05T13:30:58.235Z [INFO] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: election won: term=2 tally=2 -2024-07-05T13:30:58.235Z [INFO] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: entering leader state: leader="Node at ab2efbb1-f3f9-e20b-5960-a09c2f2f9104 [Leader]" -2024-07-05T13:30:58.235Z [INFO] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: added peer, starting replication: peer=server-45e32979-22cd-bf0d-941c-17bfc220c7f1 -2024-07-05T13:30:58.235Z [INFO] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: pipelining replication: peer="{Voter server-45e32979-22cd-bf0d-941c-17bfc220c7f1 45e32979-22cd-bf0d-941c-17bfc220c7f1}" -2024-07-05T13:30:58.340Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:58.235237836 +0000 UTC m=+99.005148551 (1 nodes), 52.789188ms from start of poll, 53.229293ms from cluster start. Timeout at 2024-07-05 13:30:58.340461024 +0000 UTC m=+99.110371699, 105.223148ms after stability -2024-07-05T13:30:58.347Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:30:58.453Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:30:58.347215107 +0000 UTC m=+99.117125822 (1 nodes), 1.44µs from start of poll, 165.206564ms from cluster start. Timeout at 2024-07-05 13:30:58.452969117 +0000 UTC m=+99.222879792, 105.75397ms after stability - raft_test.go:1799: [INFO] Disconnecting Node at 45e32979-22cd-bf0d-941c-17bfc220c7f1 [Follower] -2024-07-05T13:30:58.456Z [ERROR] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: failed to heartbeat to: peer=45e32979-22cd-bf0d-941c-17bfc220c7f1 backoff time=10ms error="failed to connect to peer: 45e32979-22cd-bf0d-941c-17bfc220c7f1" -2024-07-05T13:30:58.461Z [ERROR] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: failed to pipeline appendEntries: peer="{Voter server-45e32979-22cd-bf0d-941c-17bfc220c7f1 45e32979-22cd-bf0d-941c-17bfc220c7f1}" error="append pipeline closed" -2024-07-05T13:30:58.461Z [INFO] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: aborting pipeline replication: peer="{Voter server-45e32979-22cd-bf0d-941c-17bfc220c7f1 45e32979-22cd-bf0d-941c-17bfc220c7f1}" -2024-07-05T13:30:58.469Z [ERROR] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: failed to appendEntries to: peer="{Voter server-45e32979-22cd-bf0d-941c-17bfc220c7f1 45e32979-22cd-bf0d-941c-17bfc220c7f1}" error="failed to connect to peer: 45e32979-22cd-bf0d-941c-17bfc220c7f1" -2024-07-05T13:30:58.472Z [ERROR] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: failed to heartbeat to: peer=45e32979-22cd-bf0d-941c-17bfc220c7f1 backoff time=10ms error="failed to connect to peer: 45e32979-22cd-bf0d-941c-17bfc220c7f1" -2024-07-05T13:30:58.485Z [ERROR] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: failed to appendEntries to: peer="{Voter server-45e32979-22cd-bf0d-941c-17bfc220c7f1 45e32979-22cd-bf0d-941c-17bfc220c7f1}" error="failed to connect to peer: 45e32979-22cd-bf0d-941c-17bfc220c7f1" -2024-07-05T13:30:58.488Z [ERROR] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: failed to heartbeat to: peer=45e32979-22cd-bf0d-941c-17bfc220c7f1 backoff time=10ms error="failed to connect to peer: 45e32979-22cd-bf0d-941c-17bfc220c7f1" -2024-07-05T13:30:58.502Z [ERROR] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: failed to appendEntries to: peer="{Voter server-45e32979-22cd-bf0d-941c-17bfc220c7f1 45e32979-22cd-bf0d-941c-17bfc220c7f1}" error="failed to connect to peer: 45e32979-22cd-bf0d-941c-17bfc220c7f1" -2024-07-05T13:30:58.502Z [WARN] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: failed to contact: server-id=server-45e32979-22cd-bf0d-941c-17bfc220c7f1 time=50.050244ms -2024-07-05T13:30:58.503Z [WARN] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: failed to contact quorum of nodes, stepping down -2024-07-05T13:30:58.503Z [INFO] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: entering follower state: follower="Node at ab2efbb1-f3f9-e20b-5960-a09c2f2f9104 [Follower]" leader-address= leader-id= -2024-07-05T13:30:58.503Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:58.511Z [WARN] server-45e32979-22cd-bf0d-941c-17bfc220c7f1: heartbeat timeout reached, starting election: last-leader-addr=ab2efbb1-f3f9-e20b-5960-a09c2f2f9104 last-leader-id=server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104 -2024-07-05T13:30:58.512Z [INFO] server-45e32979-22cd-bf0d-941c-17bfc220c7f1: entering candidate state: node="Node at 45e32979-22cd-bf0d-941c-17bfc220c7f1 [Candidate]" term=3 -2024-07-05T13:30:58.512Z [ERROR] server-45e32979-22cd-bf0d-941c-17bfc220c7f1: failed to make requestVote RPC: target="{Voter server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104 ab2efbb1-f3f9-e20b-5960-a09c2f2f9104}" error="failed to connect to peer: ab2efbb1-f3f9-e20b-5960-a09c2f2f9104" term=3 -2024-07-05T13:30:58.580Z [WARN] server-45e32979-22cd-bf0d-941c-17bfc220c7f1: Election timeout reached, restarting election -2024-07-05T13:30:58.580Z [INFO] server-45e32979-22cd-bf0d-941c-17bfc220c7f1: entering candidate state: node="Node at 45e32979-22cd-bf0d-941c-17bfc220c7f1 [Candidate]" term=4 -2024-07-05T13:30:58.580Z [ERROR] server-45e32979-22cd-bf0d-941c-17bfc220c7f1: failed to make requestVote RPC: target="{Voter server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104 ab2efbb1-f3f9-e20b-5960-a09c2f2f9104}" error="failed to connect to peer: ab2efbb1-f3f9-e20b-5960-a09c2f2f9104" term=4 -2024-07-05T13:30:58.587Z [WARN] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:58.587Z [INFO] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: entering candidate state: node="Node at ab2efbb1-f3f9-e20b-5960-a09c2f2f9104 [Candidate]" term=3 -2024-07-05T13:30:58.587Z [ERROR] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: failed to make requestVote RPC: target="{Voter server-45e32979-22cd-bf0d-941c-17bfc220c7f1 45e32979-22cd-bf0d-941c-17bfc220c7f1}" error="failed to connect to peer: 45e32979-22cd-bf0d-941c-17bfc220c7f1" term=3 -2024-07-05T13:30:58.647Z [WARN] server-45e32979-22cd-bf0d-941c-17bfc220c7f1: Election timeout reached, restarting election -2024-07-05T13:30:58.647Z [INFO] server-45e32979-22cd-bf0d-941c-17bfc220c7f1: entering candidate state: node="Node at 45e32979-22cd-bf0d-941c-17bfc220c7f1 [Candidate]" term=5 -2024-07-05T13:30:58.647Z [ERROR] server-45e32979-22cd-bf0d-941c-17bfc220c7f1: failed to make requestVote RPC: target="{Voter server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104 ab2efbb1-f3f9-e20b-5960-a09c2f2f9104}" error="failed to connect to peer: ab2efbb1-f3f9-e20b-5960-a09c2f2f9104" term=5 -2024-07-05T13:30:58.655Z [WARN] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: Election timeout reached, restarting election -2024-07-05T13:30:58.655Z [INFO] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: entering candidate state: node="Node at ab2efbb1-f3f9-e20b-5960-a09c2f2f9104 [Candidate]" term=4 -2024-07-05T13:30:58.655Z [ERROR] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: failed to make requestVote RPC: target="{Voter server-45e32979-22cd-bf0d-941c-17bfc220c7f1 45e32979-22cd-bf0d-941c-17bfc220c7f1}" error="failed to connect to peer: 45e32979-22cd-bf0d-941c-17bfc220c7f1" term=4 -2024-07-05T13:30:58.693Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:58.587536876 +0000 UTC m=+99.357447551 (0 nodes), 84.38856ms from start of poll, 405.528293ms from cluster start. Timeout at 2024-07-05 13:30:58.693330364 +0000 UTC m=+99.463241039, 105.793488ms after stability -2024-07-05T13:30:58.740Z [WARN] server-45e32979-22cd-bf0d-941c-17bfc220c7f1: Election timeout reached, restarting election -2024-07-05T13:30:58.740Z [INFO] server-45e32979-22cd-bf0d-941c-17bfc220c7f1: entering candidate state: node="Node at 45e32979-22cd-bf0d-941c-17bfc220c7f1 [Candidate]" term=6 -2024-07-05T13:30:58.740Z [ERROR] server-45e32979-22cd-bf0d-941c-17bfc220c7f1: failed to make requestVote RPC: target="{Voter server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104 ab2efbb1-f3f9-e20b-5960-a09c2f2f9104}" error="failed to connect to peer: ab2efbb1-f3f9-e20b-5960-a09c2f2f9104" term=6 -2024-07-05T13:30:58.749Z [WARN] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: Election timeout reached, restarting election -2024-07-05T13:30:58.749Z [INFO] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: entering candidate state: node="Node at ab2efbb1-f3f9-e20b-5960-a09c2f2f9104 [Candidate]" term=5 -2024-07-05T13:30:58.750Z [ERROR] server-ab2efbb1-f3f9-e20b-5960-a09c2f2f9104: failed to make requestVote RPC: target="{Voter server-45e32979-22cd-bf0d-941c-17bfc220c7f1 45e32979-22cd-bf0d-941c-17bfc220c7f1}" error="failed to connect to peer: 45e32979-22cd-bf0d-941c-17bfc220c7f1" term=5 ---- PASS: TestRaft_LeaderLeaseExpire (0.62s) +2024-07-05T13:37:14.849Z [INFO] server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63 Address:616eba95-d79c-c9dd-9e05-abe5ce4b8b63} {Suffrage:Voter ID:server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec Address:ceb17830-cc10-aaa9-cc16-b82ef5bb8fec}]" +2024-07-05T13:37:14.850Z [INFO] server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63: entering follower state: follower="Node at 616eba95-d79c-c9dd-9e05-abe5ce4b8b63 [Follower]" leader-address= leader-id= +2024-07-05T13:37:14.850Z [INFO] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63 Address:616eba95-d79c-c9dd-9e05-abe5ce4b8b63} {Suffrage:Voter ID:server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec Address:ceb17830-cc10-aaa9-cc16-b82ef5bb8fec}]" +2024-07-05T13:37:14.850Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:14.850Z [INFO] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: entering follower state: follower="Node at ceb17830-cc10-aaa9-cc16-b82ef5bb8fec [Follower]" leader-address= leader-id= +2024-07-05T13:37:14.933Z [WARN] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:14.934Z [INFO] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: entering candidate state: node="Node at ceb17830-cc10-aaa9-cc16-b82ef5bb8fec [Candidate]" term=2 +2024-07-05T13:37:14.935Z [INFO] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: election won: term=2 tally=2 +2024-07-05T13:37:14.935Z [INFO] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: entering leader state: leader="Node at ceb17830-cc10-aaa9-cc16-b82ef5bb8fec [Leader]" +2024-07-05T13:37:14.935Z [INFO] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: added peer, starting replication: peer=server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63 +2024-07-05T13:37:14.936Z [INFO] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: pipelining replication: peer="{Voter server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63 616eba95-d79c-c9dd-9e05-abe5ce4b8b63}" +2024-07-05T13:37:15.041Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:14.93641985 +0000 UTC m=+100.932340718 (1 nodes), 85.949389ms from start of poll, 87.279948ms from cluster start. Timeout at 2024-07-05 13:37:15.041456724 +0000 UTC m=+101.037378175, 105.037457ms after stability +2024-07-05T13:37:15.047Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:15.153Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:15.047350616 +0000 UTC m=+101.043271776 (1 nodes), 6.125µs from start of poll, 198.211006ms from cluster start. Timeout at 2024-07-05 13:37:15.152948063 +0000 UTC m=+101.148869514, 105.597738ms after stability + raft_test.go:1799: [INFO] Disconnecting Node at 616eba95-d79c-c9dd-9e05-abe5ce4b8b63 [Follower] +2024-07-05T13:37:15.158Z [ERROR] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: failed to pipeline appendEntries: peer="{Voter server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63 616eba95-d79c-c9dd-9e05-abe5ce4b8b63}" error="append pipeline closed" +2024-07-05T13:37:15.158Z [INFO] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: aborting pipeline replication: peer="{Voter server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63 616eba95-d79c-c9dd-9e05-abe5ce4b8b63}" +2024-07-05T13:37:15.158Z [ERROR] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: failed to heartbeat to: peer=616eba95-d79c-c9dd-9e05-abe5ce4b8b63 backoff time=10ms error="failed to connect to peer: 616eba95-d79c-c9dd-9e05-abe5ce4b8b63" +2024-07-05T13:37:15.164Z [ERROR] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: failed to appendEntries to: peer="{Voter server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63 616eba95-d79c-c9dd-9e05-abe5ce4b8b63}" error="failed to connect to peer: 616eba95-d79c-c9dd-9e05-abe5ce4b8b63" +2024-07-05T13:37:15.180Z [ERROR] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: failed to heartbeat to: peer=616eba95-d79c-c9dd-9e05-abe5ce4b8b63 backoff time=10ms error="failed to connect to peer: 616eba95-d79c-c9dd-9e05-abe5ce4b8b63" +2024-07-05T13:37:15.181Z [ERROR] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: failed to appendEntries to: peer="{Voter server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63 616eba95-d79c-c9dd-9e05-abe5ce4b8b63}" error="failed to connect to peer: 616eba95-d79c-c9dd-9e05-abe5ce4b8b63" +2024-07-05T13:37:15.199Z [ERROR] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: failed to heartbeat to: peer=616eba95-d79c-c9dd-9e05-abe5ce4b8b63 backoff time=10ms error="failed to connect to peer: 616eba95-d79c-c9dd-9e05-abe5ce4b8b63" +2024-07-05T13:37:15.201Z [WARN] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: failed to contact: server-id=server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63 time=50.133668ms +2024-07-05T13:37:15.201Z [WARN] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: failed to contact quorum of nodes, stepping down +2024-07-05T13:37:15.201Z [ERROR] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: failed to appendEntries to: peer="{Voter server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63 616eba95-d79c-c9dd-9e05-abe5ce4b8b63}" error="failed to connect to peer: 616eba95-d79c-c9dd-9e05-abe5ce4b8b63" +2024-07-05T13:37:15.201Z [INFO] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: entering follower state: follower="Node at ceb17830-cc10-aaa9-cc16-b82ef5bb8fec [Follower]" leader-address= leader-id= +2024-07-05T13:37:15.202Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:15.247Z [WARN] server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63: heartbeat timeout reached, starting election: last-leader-addr=ceb17830-cc10-aaa9-cc16-b82ef5bb8fec last-leader-id=server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec +2024-07-05T13:37:15.247Z [INFO] server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63: entering candidate state: node="Node at 616eba95-d79c-c9dd-9e05-abe5ce4b8b63 [Candidate]" term=3 +2024-07-05T13:37:15.248Z [ERROR] server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63: failed to make requestVote RPC: target="{Voter server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec ceb17830-cc10-aaa9-cc16-b82ef5bb8fec}" error="failed to connect to peer: ceb17830-cc10-aaa9-cc16-b82ef5bb8fec" term=3 +2024-07-05T13:37:15.266Z [WARN] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:15.266Z [INFO] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: entering candidate state: node="Node at ceb17830-cc10-aaa9-cc16-b82ef5bb8fec [Candidate]" term=3 +2024-07-05T13:37:15.266Z [ERROR] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: failed to make requestVote RPC: target="{Voter server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63 616eba95-d79c-c9dd-9e05-abe5ce4b8b63}" error="failed to connect to peer: 616eba95-d79c-c9dd-9e05-abe5ce4b8b63" term=3 +2024-07-05T13:37:15.346Z [WARN] server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63: Election timeout reached, restarting election +2024-07-05T13:37:15.347Z [INFO] server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63: entering candidate state: node="Node at 616eba95-d79c-c9dd-9e05-abe5ce4b8b63 [Candidate]" term=4 +2024-07-05T13:37:15.347Z [ERROR] server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63: failed to make requestVote RPC: target="{Voter server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec ceb17830-cc10-aaa9-cc16-b82ef5bb8fec}" error="failed to connect to peer: ceb17830-cc10-aaa9-cc16-b82ef5bb8fec" term=4 +2024-07-05T13:37:15.357Z [WARN] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: Election timeout reached, restarting election +2024-07-05T13:37:15.357Z [INFO] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: entering candidate state: node="Node at ceb17830-cc10-aaa9-cc16-b82ef5bb8fec [Candidate]" term=4 +2024-07-05T13:37:15.358Z [ERROR] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: failed to make requestVote RPC: target="{Voter server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63 616eba95-d79c-c9dd-9e05-abe5ce4b8b63}" error="failed to connect to peer: 616eba95-d79c-c9dd-9e05-abe5ce4b8b63" term=4 +2024-07-05T13:37:15.372Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:15.266657778 +0000 UTC m=+101.262578938 (0 nodes), 64.55253ms from start of poll, 417.518168ms from cluster start. Timeout at 2024-07-05 13:37:15.371739859 +0000 UTC m=+101.367661311, 105.082373ms after stability +2024-07-05T13:37:15.410Z [WARN] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: Election timeout reached, restarting election +2024-07-05T13:37:15.412Z [INFO] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: entering candidate state: node="Node at ceb17830-cc10-aaa9-cc16-b82ef5bb8fec [Candidate]" term=5 +2024-07-05T13:37:15.422Z [ERROR] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: failed to make requestVote RPC: target="{Voter server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63 616eba95-d79c-c9dd-9e05-abe5ce4b8b63}" error="failed to connect to peer: 616eba95-d79c-c9dd-9e05-abe5ce4b8b63" term=5 +2024-07-05T13:37:15.435Z [WARN] server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63: Election timeout reached, restarting election +2024-07-05T13:37:15.435Z [INFO] server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63: entering candidate state: node="Node at 616eba95-d79c-c9dd-9e05-abe5ce4b8b63 [Candidate]" term=5 +2024-07-05T13:37:15.435Z [ERROR] server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63: failed to make requestVote RPC: target="{Voter server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec ceb17830-cc10-aaa9-cc16-b82ef5bb8fec}" error="failed to connect to peer: ceb17830-cc10-aaa9-cc16-b82ef5bb8fec" term=5 +2024-07-05T13:37:15.466Z [WARN] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: Election timeout reached, restarting election +2024-07-05T13:37:15.466Z [INFO] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: entering candidate state: node="Node at ceb17830-cc10-aaa9-cc16-b82ef5bb8fec [Candidate]" term=6 +2024-07-05T13:37:15.467Z [ERROR] server-ceb17830-cc10-aaa9-cc16-b82ef5bb8fec: failed to make requestVote RPC: target="{Voter server-616eba95-d79c-c9dd-9e05-abe5ce4b8b63 616eba95-d79c-c9dd-9e05-abe5ce4b8b63}" error="failed to connect to peer: 616eba95-d79c-c9dd-9e05-abe5ce4b8b63" term=6 +--- PASS: TestRaft_LeaderLeaseExpire (0.64s) === RUN TestRaft_Barrier -2024-07-05T13:30:58.800Z [INFO] server-c7a6580c-2789-1c29-9efc-b4cfb86c1307: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7a6580c-2789-1c29-9efc-b4cfb86c1307 Address:c7a6580c-2789-1c29-9efc-b4cfb86c1307} {Suffrage:Voter ID:server-d2f02d40-0509-ab18-7077-689c4de836c5 Address:d2f02d40-0509-ab18-7077-689c4de836c5} {Suffrage:Voter ID:server-dc324116-b6ab-85c3-13f5-60f3a8f469b6 Address:dc324116-b6ab-85c3-13f5-60f3a8f469b6}]" -2024-07-05T13:30:58.800Z [INFO] server-d2f02d40-0509-ab18-7077-689c4de836c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7a6580c-2789-1c29-9efc-b4cfb86c1307 Address:c7a6580c-2789-1c29-9efc-b4cfb86c1307} {Suffrage:Voter ID:server-d2f02d40-0509-ab18-7077-689c4de836c5 Address:d2f02d40-0509-ab18-7077-689c4de836c5} {Suffrage:Voter ID:server-dc324116-b6ab-85c3-13f5-60f3a8f469b6 Address:dc324116-b6ab-85c3-13f5-60f3a8f469b6}]" -2024-07-05T13:30:58.801Z [INFO] server-dc324116-b6ab-85c3-13f5-60f3a8f469b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7a6580c-2789-1c29-9efc-b4cfb86c1307 Address:c7a6580c-2789-1c29-9efc-b4cfb86c1307} {Suffrage:Voter ID:server-d2f02d40-0509-ab18-7077-689c4de836c5 Address:d2f02d40-0509-ab18-7077-689c4de836c5} {Suffrage:Voter ID:server-dc324116-b6ab-85c3-13f5-60f3a8f469b6 Address:dc324116-b6ab-85c3-13f5-60f3a8f469b6}]" -2024-07-05T13:30:58.801Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:58.801Z [INFO] server-c7a6580c-2789-1c29-9efc-b4cfb86c1307: entering follower state: follower="Node at c7a6580c-2789-1c29-9efc-b4cfb86c1307 [Follower]" leader-address= leader-id= -2024-07-05T13:30:58.801Z [INFO] server-d2f02d40-0509-ab18-7077-689c4de836c5: entering follower state: follower="Node at d2f02d40-0509-ab18-7077-689c4de836c5 [Follower]" leader-address= leader-id= -2024-07-05T13:30:58.801Z [INFO] server-dc324116-b6ab-85c3-13f5-60f3a8f469b6: entering follower state: follower="Node at dc324116-b6ab-85c3-13f5-60f3a8f469b6 [Follower]" leader-address= leader-id= -2024-07-05T13:30:58.870Z [WARN] server-c7a6580c-2789-1c29-9efc-b4cfb86c1307: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:58.870Z [INFO] server-c7a6580c-2789-1c29-9efc-b4cfb86c1307: entering candidate state: node="Node at c7a6580c-2789-1c29-9efc-b4cfb86c1307 [Candidate]" term=2 -2024-07-05T13:30:58.870Z [INFO] server-c7a6580c-2789-1c29-9efc-b4cfb86c1307: election won: term=2 tally=2 -2024-07-05T13:30:58.870Z [INFO] server-c7a6580c-2789-1c29-9efc-b4cfb86c1307: entering leader state: leader="Node at c7a6580c-2789-1c29-9efc-b4cfb86c1307 [Leader]" -2024-07-05T13:30:58.871Z [INFO] server-c7a6580c-2789-1c29-9efc-b4cfb86c1307: added peer, starting replication: peer=server-d2f02d40-0509-ab18-7077-689c4de836c5 -2024-07-05T13:30:58.871Z [INFO] server-c7a6580c-2789-1c29-9efc-b4cfb86c1307: added peer, starting replication: peer=server-dc324116-b6ab-85c3-13f5-60f3a8f469b6 -2024-07-05T13:30:58.871Z [INFO] server-c7a6580c-2789-1c29-9efc-b4cfb86c1307: pipelining replication: peer="{Voter server-d2f02d40-0509-ab18-7077-689c4de836c5 d2f02d40-0509-ab18-7077-689c4de836c5}" -2024-07-05T13:30:58.871Z [INFO] server-c7a6580c-2789-1c29-9efc-b4cfb86c1307: pipelining replication: peer="{Voter server-dc324116-b6ab-85c3-13f5-60f3a8f469b6 dc324116-b6ab-85c3-13f5-60f3a8f469b6}" -2024-07-05T13:30:58.984Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:58.872488278 +0000 UTC m=+99.642398953 (1 nodes), 71.25894ms from start of poll, 71.919597ms from cluster start. Timeout at 2024-07-05 13:30:58.9841738 +0000 UTC m=+99.754084475, 111.685522ms after stability -2024-07-05T13:30:59.016Z [INFO] server-c7a6580c-2789-1c29-9efc-b4cfb86c1307: aborting pipeline replication: peer="{Voter server-dc324116-b6ab-85c3-13f5-60f3a8f469b6 dc324116-b6ab-85c3-13f5-60f3a8f469b6}" -2024-07-05T13:30:59.016Z [INFO] server-c7a6580c-2789-1c29-9efc-b4cfb86c1307: aborting pipeline replication: peer="{Voter server-d2f02d40-0509-ab18-7077-689c4de836c5 d2f02d40-0509-ab18-7077-689c4de836c5}" ---- PASS: TestRaft_Barrier (0.22s) +2024-07-05T13:37:15.485Z [INFO] server-a3f5ee5f-45f8-7f42-1e3b-44ac5e8b9edc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3f5ee5f-45f8-7f42-1e3b-44ac5e8b9edc Address:a3f5ee5f-45f8-7f42-1e3b-44ac5e8b9edc} {Suffrage:Voter ID:server-017803d4-58e2-dde3-45c0-fffaeacbe18a Address:017803d4-58e2-dde3-45c0-fffaeacbe18a} {Suffrage:Voter ID:server-1c2bb100-5b36-fe3f-2fa9-e7a3aae8d456 Address:1c2bb100-5b36-fe3f-2fa9-e7a3aae8d456}]" +2024-07-05T13:37:15.486Z [INFO] server-a3f5ee5f-45f8-7f42-1e3b-44ac5e8b9edc: entering follower state: follower="Node at a3f5ee5f-45f8-7f42-1e3b-44ac5e8b9edc [Follower]" leader-address= leader-id= +2024-07-05T13:37:15.486Z [INFO] server-017803d4-58e2-dde3-45c0-fffaeacbe18a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3f5ee5f-45f8-7f42-1e3b-44ac5e8b9edc Address:a3f5ee5f-45f8-7f42-1e3b-44ac5e8b9edc} {Suffrage:Voter ID:server-017803d4-58e2-dde3-45c0-fffaeacbe18a Address:017803d4-58e2-dde3-45c0-fffaeacbe18a} {Suffrage:Voter ID:server-1c2bb100-5b36-fe3f-2fa9-e7a3aae8d456 Address:1c2bb100-5b36-fe3f-2fa9-e7a3aae8d456}]" +2024-07-05T13:37:15.487Z [INFO] server-017803d4-58e2-dde3-45c0-fffaeacbe18a: entering follower state: follower="Node at 017803d4-58e2-dde3-45c0-fffaeacbe18a [Follower]" leader-address= leader-id= +2024-07-05T13:37:15.487Z [INFO] server-1c2bb100-5b36-fe3f-2fa9-e7a3aae8d456: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3f5ee5f-45f8-7f42-1e3b-44ac5e8b9edc Address:a3f5ee5f-45f8-7f42-1e3b-44ac5e8b9edc} {Suffrage:Voter ID:server-017803d4-58e2-dde3-45c0-fffaeacbe18a Address:017803d4-58e2-dde3-45c0-fffaeacbe18a} {Suffrage:Voter ID:server-1c2bb100-5b36-fe3f-2fa9-e7a3aae8d456 Address:1c2bb100-5b36-fe3f-2fa9-e7a3aae8d456}]" +2024-07-05T13:37:15.487Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:15.487Z [INFO] server-1c2bb100-5b36-fe3f-2fa9-e7a3aae8d456: entering follower state: follower="Node at 1c2bb100-5b36-fe3f-2fa9-e7a3aae8d456 [Follower]" leader-address= leader-id= +2024-07-05T13:37:15.540Z [WARN] server-017803d4-58e2-dde3-45c0-fffaeacbe18a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:15.540Z [INFO] server-017803d4-58e2-dde3-45c0-fffaeacbe18a: entering candidate state: node="Node at 017803d4-58e2-dde3-45c0-fffaeacbe18a [Candidate]" term=2 +2024-07-05T13:37:15.541Z [INFO] server-017803d4-58e2-dde3-45c0-fffaeacbe18a: election won: term=2 tally=2 +2024-07-05T13:37:15.541Z [INFO] server-017803d4-58e2-dde3-45c0-fffaeacbe18a: entering leader state: leader="Node at 017803d4-58e2-dde3-45c0-fffaeacbe18a [Leader]" +2024-07-05T13:37:15.541Z [INFO] server-017803d4-58e2-dde3-45c0-fffaeacbe18a: added peer, starting replication: peer=server-a3f5ee5f-45f8-7f42-1e3b-44ac5e8b9edc +2024-07-05T13:37:15.542Z [INFO] server-017803d4-58e2-dde3-45c0-fffaeacbe18a: added peer, starting replication: peer=server-1c2bb100-5b36-fe3f-2fa9-e7a3aae8d456 +2024-07-05T13:37:15.542Z [INFO] server-017803d4-58e2-dde3-45c0-fffaeacbe18a: pipelining replication: peer="{Voter server-1c2bb100-5b36-fe3f-2fa9-e7a3aae8d456 1c2bb100-5b36-fe3f-2fa9-e7a3aae8d456}" +2024-07-05T13:37:15.543Z [INFO] server-017803d4-58e2-dde3-45c0-fffaeacbe18a: pipelining replication: peer="{Voter server-a3f5ee5f-45f8-7f42-1e3b-44ac5e8b9edc a3f5ee5f-45f8-7f42-1e3b-44ac5e8b9edc}" +2024-07-05T13:37:15.647Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:15.541695297 +0000 UTC m=+101.537617040 (1 nodes), 54.329216ms from start of poll, 56.569175ms from cluster start. Timeout at 2024-07-05 13:37:15.646931959 +0000 UTC m=+101.642853119, 105.236079ms after stability +2024-07-05T13:37:15.711Z [INFO] server-017803d4-58e2-dde3-45c0-fffaeacbe18a: aborting pipeline replication: peer="{Voter server-a3f5ee5f-45f8-7f42-1e3b-44ac5e8b9edc a3f5ee5f-45f8-7f42-1e3b-44ac5e8b9edc}" +2024-07-05T13:37:15.711Z [INFO] server-017803d4-58e2-dde3-45c0-fffaeacbe18a: aborting pipeline replication: peer="{Voter server-1c2bb100-5b36-fe3f-2fa9-e7a3aae8d456 1c2bb100-5b36-fe3f-2fa9-e7a3aae8d456}" +2024-07-05T13:37:16.212Z [ERROR] server-017803d4-58e2-dde3-45c0-fffaeacbe18a: failed to heartbeat to: peer=1c2bb100-5b36-fe3f-2fa9-e7a3aae8d456 backoff time=10ms error="command timed out" +--- PASS: TestRaft_Barrier (0.74s) === RUN TestRaft_VerifyLeader -2024-07-05T13:30:59.018Z [INFO] server-e0b2af52-3935-4bc3-fbb4-370a7d0e2e55: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0b2af52-3935-4bc3-fbb4-370a7d0e2e55 Address:e0b2af52-3935-4bc3-fbb4-370a7d0e2e55} {Suffrage:Voter ID:server-b17e7a7a-ef2e-046c-6c8d-8b39dadf383f Address:b17e7a7a-ef2e-046c-6c8d-8b39dadf383f} {Suffrage:Voter ID:server-204eb38e-b222-607a-a380-19537f96571c Address:204eb38e-b222-607a-a380-19537f96571c}]" -2024-07-05T13:30:59.019Z [INFO] server-e0b2af52-3935-4bc3-fbb4-370a7d0e2e55: entering follower state: follower="Node at e0b2af52-3935-4bc3-fbb4-370a7d0e2e55 [Follower]" leader-address= leader-id= -2024-07-05T13:30:59.019Z [INFO] server-b17e7a7a-ef2e-046c-6c8d-8b39dadf383f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0b2af52-3935-4bc3-fbb4-370a7d0e2e55 Address:e0b2af52-3935-4bc3-fbb4-370a7d0e2e55} {Suffrage:Voter ID:server-b17e7a7a-ef2e-046c-6c8d-8b39dadf383f Address:b17e7a7a-ef2e-046c-6c8d-8b39dadf383f} {Suffrage:Voter ID:server-204eb38e-b222-607a-a380-19537f96571c Address:204eb38e-b222-607a-a380-19537f96571c}]" -2024-07-05T13:30:59.019Z [INFO] server-b17e7a7a-ef2e-046c-6c8d-8b39dadf383f: entering follower state: follower="Node at b17e7a7a-ef2e-046c-6c8d-8b39dadf383f [Follower]" leader-address= leader-id= -2024-07-05T13:30:59.019Z [INFO] server-204eb38e-b222-607a-a380-19537f96571c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0b2af52-3935-4bc3-fbb4-370a7d0e2e55 Address:e0b2af52-3935-4bc3-fbb4-370a7d0e2e55} {Suffrage:Voter ID:server-b17e7a7a-ef2e-046c-6c8d-8b39dadf383f Address:b17e7a7a-ef2e-046c-6c8d-8b39dadf383f} {Suffrage:Voter ID:server-204eb38e-b222-607a-a380-19537f96571c Address:204eb38e-b222-607a-a380-19537f96571c}]" -2024-07-05T13:30:59.019Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:59.019Z [INFO] server-204eb38e-b222-607a-a380-19537f96571c: entering follower state: follower="Node at 204eb38e-b222-607a-a380-19537f96571c [Follower]" leader-address= leader-id= -2024-07-05T13:30:59.076Z [WARN] server-b17e7a7a-ef2e-046c-6c8d-8b39dadf383f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:59.077Z [INFO] server-b17e7a7a-ef2e-046c-6c8d-8b39dadf383f: entering candidate state: node="Node at b17e7a7a-ef2e-046c-6c8d-8b39dadf383f [Candidate]" term=2 -2024-07-05T13:30:59.077Z [INFO] server-b17e7a7a-ef2e-046c-6c8d-8b39dadf383f: election won: term=2 tally=2 -2024-07-05T13:30:59.077Z [INFO] server-b17e7a7a-ef2e-046c-6c8d-8b39dadf383f: entering leader state: leader="Node at b17e7a7a-ef2e-046c-6c8d-8b39dadf383f [Leader]" -2024-07-05T13:30:59.077Z [INFO] server-b17e7a7a-ef2e-046c-6c8d-8b39dadf383f: added peer, starting replication: peer=server-e0b2af52-3935-4bc3-fbb4-370a7d0e2e55 -2024-07-05T13:30:59.077Z [INFO] server-b17e7a7a-ef2e-046c-6c8d-8b39dadf383f: added peer, starting replication: peer=server-204eb38e-b222-607a-a380-19537f96571c -2024-07-05T13:30:59.077Z [INFO] server-b17e7a7a-ef2e-046c-6c8d-8b39dadf383f: pipelining replication: peer="{Voter server-204eb38e-b222-607a-a380-19537f96571c 204eb38e-b222-607a-a380-19537f96571c}" -2024-07-05T13:30:59.077Z [WARN] server-204eb38e-b222-607a-a380-19537f96571c: rejecting vote request since our last term is greater: candidate=b17e7a7a-ef2e-046c-6c8d-8b39dadf383f last-term=2 last-candidate-term=1 -2024-07-05T13:30:59.077Z [INFO] server-b17e7a7a-ef2e-046c-6c8d-8b39dadf383f: pipelining replication: peer="{Voter server-e0b2af52-3935-4bc3-fbb4-370a7d0e2e55 e0b2af52-3935-4bc3-fbb4-370a7d0e2e55}" -2024-07-05T13:30:59.183Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:59.077818994 +0000 UTC m=+99.847729669 (1 nodes), 58.373552ms from start of poll, 59.105606ms from cluster start. Timeout at 2024-07-05 13:30:59.183774517 +0000 UTC m=+99.953685152, 105.955483ms after stability -2024-07-05T13:30:59.184Z [INFO] server-b17e7a7a-ef2e-046c-6c8d-8b39dadf383f: aborting pipeline replication: peer="{Voter server-204eb38e-b222-607a-a380-19537f96571c 204eb38e-b222-607a-a380-19537f96571c}" -2024-07-05T13:30:59.184Z [INFO] server-b17e7a7a-ef2e-046c-6c8d-8b39dadf383f: aborting pipeline replication: peer="{Voter server-e0b2af52-3935-4bc3-fbb4-370a7d0e2e55 e0b2af52-3935-4bc3-fbb4-370a7d0e2e55}" -2024-07-05T13:30:59.684Z [ERROR] server-b17e7a7a-ef2e-046c-6c8d-8b39dadf383f: failed to heartbeat to: peer=e0b2af52-3935-4bc3-fbb4-370a7d0e2e55 backoff time=10ms error="command timed out" ---- PASS: TestRaft_VerifyLeader (0.67s) +2024-07-05T13:37:16.223Z [INFO] server-9334e378-67f5-21ca-e95d-31d213414cd2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9334e378-67f5-21ca-e95d-31d213414cd2 Address:9334e378-67f5-21ca-e95d-31d213414cd2} {Suffrage:Voter ID:server-36b8ab78-0e5f-0d0c-c363-ee132a89043e Address:36b8ab78-0e5f-0d0c-c363-ee132a89043e} {Suffrage:Voter ID:server-6fb6b22c-6f53-2697-5c18-d79743d8b69c Address:6fb6b22c-6f53-2697-5c18-d79743d8b69c}]" +2024-07-05T13:37:16.224Z [INFO] server-9334e378-67f5-21ca-e95d-31d213414cd2: entering follower state: follower="Node at 9334e378-67f5-21ca-e95d-31d213414cd2 [Follower]" leader-address= leader-id= +2024-07-05T13:37:16.224Z [INFO] server-36b8ab78-0e5f-0d0c-c363-ee132a89043e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9334e378-67f5-21ca-e95d-31d213414cd2 Address:9334e378-67f5-21ca-e95d-31d213414cd2} {Suffrage:Voter ID:server-36b8ab78-0e5f-0d0c-c363-ee132a89043e Address:36b8ab78-0e5f-0d0c-c363-ee132a89043e} {Suffrage:Voter ID:server-6fb6b22c-6f53-2697-5c18-d79743d8b69c Address:6fb6b22c-6f53-2697-5c18-d79743d8b69c}]" +2024-07-05T13:37:16.225Z [INFO] server-6fb6b22c-6f53-2697-5c18-d79743d8b69c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9334e378-67f5-21ca-e95d-31d213414cd2 Address:9334e378-67f5-21ca-e95d-31d213414cd2} {Suffrage:Voter ID:server-36b8ab78-0e5f-0d0c-c363-ee132a89043e Address:36b8ab78-0e5f-0d0c-c363-ee132a89043e} {Suffrage:Voter ID:server-6fb6b22c-6f53-2697-5c18-d79743d8b69c Address:6fb6b22c-6f53-2697-5c18-d79743d8b69c}]" +2024-07-05T13:37:16.225Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:16.225Z [INFO] server-36b8ab78-0e5f-0d0c-c363-ee132a89043e: entering follower state: follower="Node at 36b8ab78-0e5f-0d0c-c363-ee132a89043e [Follower]" leader-address= leader-id= +2024-07-05T13:37:16.226Z [INFO] server-6fb6b22c-6f53-2697-5c18-d79743d8b69c: entering follower state: follower="Node at 6fb6b22c-6f53-2697-5c18-d79743d8b69c [Follower]" leader-address= leader-id= +2024-07-05T13:37:16.277Z [WARN] server-36b8ab78-0e5f-0d0c-c363-ee132a89043e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:16.277Z [INFO] server-36b8ab78-0e5f-0d0c-c363-ee132a89043e: entering candidate state: node="Node at 36b8ab78-0e5f-0d0c-c363-ee132a89043e [Candidate]" term=2 +2024-07-05T13:37:16.278Z [INFO] server-36b8ab78-0e5f-0d0c-c363-ee132a89043e: election won: term=2 tally=2 +2024-07-05T13:37:16.278Z [INFO] server-36b8ab78-0e5f-0d0c-c363-ee132a89043e: entering leader state: leader="Node at 36b8ab78-0e5f-0d0c-c363-ee132a89043e [Leader]" +2024-07-05T13:37:16.278Z [INFO] server-36b8ab78-0e5f-0d0c-c363-ee132a89043e: added peer, starting replication: peer=server-9334e378-67f5-21ca-e95d-31d213414cd2 +2024-07-05T13:37:16.278Z [INFO] server-36b8ab78-0e5f-0d0c-c363-ee132a89043e: added peer, starting replication: peer=server-6fb6b22c-6f53-2697-5c18-d79743d8b69c +2024-07-05T13:37:16.279Z [INFO] server-36b8ab78-0e5f-0d0c-c363-ee132a89043e: pipelining replication: peer="{Voter server-9334e378-67f5-21ca-e95d-31d213414cd2 9334e378-67f5-21ca-e95d-31d213414cd2}" +2024-07-05T13:37:16.279Z [INFO] server-36b8ab78-0e5f-0d0c-c363-ee132a89043e: pipelining replication: peer="{Voter server-6fb6b22c-6f53-2697-5c18-d79743d8b69c 6fb6b22c-6f53-2697-5c18-d79743d8b69c}" +2024-07-05T13:37:16.383Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:16.278414636 +0000 UTC m=+102.274335796 (1 nodes), 52.606373ms from start of poll, 55.475154ms from cluster start. Timeout at 2024-07-05 13:37:16.383689797 +0000 UTC m=+102.379610956, 105.27516ms after stability +2024-07-05T13:37:16.384Z [INFO] server-36b8ab78-0e5f-0d0c-c363-ee132a89043e: aborting pipeline replication: peer="{Voter server-6fb6b22c-6f53-2697-5c18-d79743d8b69c 6fb6b22c-6f53-2697-5c18-d79743d8b69c}" +2024-07-05T13:37:16.384Z [INFO] server-36b8ab78-0e5f-0d0c-c363-ee132a89043e: aborting pipeline replication: peer="{Voter server-9334e378-67f5-21ca-e95d-31d213414cd2 9334e378-67f5-21ca-e95d-31d213414cd2}" +--- PASS: TestRaft_VerifyLeader (0.17s) === RUN TestRaft_VerifyLeader_Single -2024-07-05T13:30:59.686Z [INFO] server-ed916171-2bbe-60dc-e09e-428b1e783886: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed916171-2bbe-60dc-e09e-428b1e783886 Address:ed916171-2bbe-60dc-e09e-428b1e783886}]" -2024-07-05T13:30:59.686Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:59.686Z [INFO] server-ed916171-2bbe-60dc-e09e-428b1e783886: entering follower state: follower="Node at ed916171-2bbe-60dc-e09e-428b1e783886 [Follower]" leader-address= leader-id= -2024-07-05T13:30:59.757Z [WARN] server-ed916171-2bbe-60dc-e09e-428b1e783886: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:59.757Z [INFO] server-ed916171-2bbe-60dc-e09e-428b1e783886: entering candidate state: node="Node at ed916171-2bbe-60dc-e09e-428b1e783886 [Candidate]" term=2 -2024-07-05T13:30:59.757Z [INFO] server-ed916171-2bbe-60dc-e09e-428b1e783886: election won: term=2 tally=1 -2024-07-05T13:30:59.757Z [INFO] server-ed916171-2bbe-60dc-e09e-428b1e783886: entering leader state: leader="Node at ed916171-2bbe-60dc-e09e-428b1e783886 [Leader]" -2024-07-05T13:30:59.865Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:59.757465709 +0000 UTC m=+100.527376384 (1 nodes), 71.168583ms from start of poll, 71.483492ms from cluster start. Timeout at 2024-07-05 13:30:59.865041015 +0000 UTC m=+100.634951690, 107.575306ms after stability +2024-07-05T13:37:16.390Z [INFO] server-00eb3ab8-3216-c3bd-3202-6a3f783b127a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00eb3ab8-3216-c3bd-3202-6a3f783b127a Address:00eb3ab8-3216-c3bd-3202-6a3f783b127a}]" +2024-07-05T13:37:16.390Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:16.390Z [INFO] server-00eb3ab8-3216-c3bd-3202-6a3f783b127a: entering follower state: follower="Node at 00eb3ab8-3216-c3bd-3202-6a3f783b127a [Follower]" leader-address= leader-id= +2024-07-05T13:37:16.455Z [WARN] server-00eb3ab8-3216-c3bd-3202-6a3f783b127a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:16.455Z [INFO] server-00eb3ab8-3216-c3bd-3202-6a3f783b127a: entering candidate state: node="Node at 00eb3ab8-3216-c3bd-3202-6a3f783b127a [Candidate]" term=2 +2024-07-05T13:37:16.455Z [INFO] server-00eb3ab8-3216-c3bd-3202-6a3f783b127a: election won: term=2 tally=1 +2024-07-05T13:37:16.455Z [INFO] server-00eb3ab8-3216-c3bd-3202-6a3f783b127a: entering leader state: leader="Node at 00eb3ab8-3216-c3bd-3202-6a3f783b127a [Leader]" +2024-07-05T13:37:16.569Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:16.463909498 +0000 UTC m=+102.459830367 (1 nodes), 73.282246ms from start of poll, 74.156063ms from cluster start. Timeout at 2024-07-05 13:37:16.569833314 +0000 UTC m=+102.565754474, 105.924107ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.18s) === RUN TestRaft_VerifyLeader_Fail -2024-07-05T13:30:59.867Z [INFO] server-88ce8027-adf4-f741-34ed-d665b5bcc0b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88ce8027-adf4-f741-34ed-d665b5bcc0b7 Address:88ce8027-adf4-f741-34ed-d665b5bcc0b7} {Suffrage:Voter ID:server-dfa32da0-c11d-9cce-f208-253a73199251 Address:dfa32da0-c11d-9cce-f208-253a73199251}]" -2024-07-05T13:30:59.867Z [INFO] server-dfa32da0-c11d-9cce-f208-253a73199251: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88ce8027-adf4-f741-34ed-d665b5bcc0b7 Address:88ce8027-adf4-f741-34ed-d665b5bcc0b7} {Suffrage:Voter ID:server-dfa32da0-c11d-9cce-f208-253a73199251 Address:dfa32da0-c11d-9cce-f208-253a73199251}]" -2024-07-05T13:30:59.867Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:30:59.867Z [INFO] server-88ce8027-adf4-f741-34ed-d665b5bcc0b7: entering follower state: follower="Node at 88ce8027-adf4-f741-34ed-d665b5bcc0b7 [Follower]" leader-address= leader-id= -2024-07-05T13:30:59.867Z [INFO] server-dfa32da0-c11d-9cce-f208-253a73199251: entering follower state: follower="Node at dfa32da0-c11d-9cce-f208-253a73199251 [Follower]" leader-address= leader-id= -2024-07-05T13:30:59.919Z [WARN] server-dfa32da0-c11d-9cce-f208-253a73199251: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:30:59.919Z [INFO] server-dfa32da0-c11d-9cce-f208-253a73199251: entering candidate state: node="Node at dfa32da0-c11d-9cce-f208-253a73199251 [Candidate]" term=2 -2024-07-05T13:30:59.919Z [INFO] server-dfa32da0-c11d-9cce-f208-253a73199251: election won: term=2 tally=2 -2024-07-05T13:30:59.920Z [INFO] server-dfa32da0-c11d-9cce-f208-253a73199251: entering leader state: leader="Node at dfa32da0-c11d-9cce-f208-253a73199251 [Leader]" -2024-07-05T13:30:59.920Z [INFO] server-dfa32da0-c11d-9cce-f208-253a73199251: added peer, starting replication: peer=server-88ce8027-adf4-f741-34ed-d665b5bcc0b7 -2024-07-05T13:30:59.920Z [INFO] server-dfa32da0-c11d-9cce-f208-253a73199251: pipelining replication: peer="{Voter server-88ce8027-adf4-f741-34ed-d665b5bcc0b7 88ce8027-adf4-f741-34ed-d665b5bcc0b7}" -2024-07-05T13:31:00.028Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:30:59.920422152 +0000 UTC m=+100.690332827 (1 nodes), 53.100617ms from start of poll, 53.57548ms from cluster start. Timeout at 2024-07-05 13:31:00.025443107 +0000 UTC m=+100.795353782, 105.020955ms after stability -2024-07-05T13:31:00.028Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:31:00.133Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:31:00.028363805 +0000 UTC m=+100.798274520 (1 nodes), 1.4µs from start of poll, 161.517173ms from cluster start. Timeout at 2024-07-05 13:31:00.133828305 +0000 UTC m=+100.903738980, 105.46446ms after stability -2024-07-05T13:31:00.135Z [ERROR] server-dfa32da0-c11d-9cce-f208-253a73199251: peer has newer term, stopping replication: peer="{Voter server-88ce8027-adf4-f741-34ed-d665b5bcc0b7 88ce8027-adf4-f741-34ed-d665b5bcc0b7}" -2024-07-05T13:31:00.136Z [INFO] server-dfa32da0-c11d-9cce-f208-253a73199251: aborting pipeline replication: peer="{Voter server-88ce8027-adf4-f741-34ed-d665b5bcc0b7 88ce8027-adf4-f741-34ed-d665b5bcc0b7}" -2024-07-05T13:31:00.136Z [INFO] server-dfa32da0-c11d-9cce-f208-253a73199251: entering follower state: follower="Node at dfa32da0-c11d-9cce-f208-253a73199251 [Follower]" leader-address= leader-id= ---- PASS: TestRaft_VerifyLeader_Fail (0.27s) +2024-07-05T13:37:16.576Z [INFO] server-6626ae65-698e-5ea0-4d01-c76e53837463: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6626ae65-698e-5ea0-4d01-c76e53837463 Address:6626ae65-698e-5ea0-4d01-c76e53837463} {Suffrage:Voter ID:server-756c7793-9921-b854-b50c-673c26c4d40c Address:756c7793-9921-b854-b50c-673c26c4d40c}]" +2024-07-05T13:37:16.576Z [INFO] server-6626ae65-698e-5ea0-4d01-c76e53837463: entering follower state: follower="Node at 6626ae65-698e-5ea0-4d01-c76e53837463 [Follower]" leader-address= leader-id= +2024-07-05T13:37:16.577Z [INFO] server-756c7793-9921-b854-b50c-673c26c4d40c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6626ae65-698e-5ea0-4d01-c76e53837463 Address:6626ae65-698e-5ea0-4d01-c76e53837463} {Suffrage:Voter ID:server-756c7793-9921-b854-b50c-673c26c4d40c Address:756c7793-9921-b854-b50c-673c26c4d40c}]" +2024-07-05T13:37:16.577Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:16.577Z [INFO] server-756c7793-9921-b854-b50c-673c26c4d40c: entering follower state: follower="Node at 756c7793-9921-b854-b50c-673c26c4d40c [Follower]" leader-address= leader-id= +2024-07-05T13:37:16.628Z [WARN] server-6626ae65-698e-5ea0-4d01-c76e53837463: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:16.628Z [INFO] server-6626ae65-698e-5ea0-4d01-c76e53837463: entering candidate state: node="Node at 6626ae65-698e-5ea0-4d01-c76e53837463 [Candidate]" term=2 +2024-07-05T13:37:16.628Z [INFO] server-6626ae65-698e-5ea0-4d01-c76e53837463: election won: term=2 tally=2 +2024-07-05T13:37:16.628Z [INFO] server-6626ae65-698e-5ea0-4d01-c76e53837463: entering leader state: leader="Node at 6626ae65-698e-5ea0-4d01-c76e53837463 [Leader]" +2024-07-05T13:37:16.628Z [INFO] server-6626ae65-698e-5ea0-4d01-c76e53837463: added peer, starting replication: peer=server-756c7793-9921-b854-b50c-673c26c4d40c +2024-07-05T13:37:16.629Z [INFO] server-6626ae65-698e-5ea0-4d01-c76e53837463: pipelining replication: peer="{Voter server-756c7793-9921-b854-b50c-673c26c4d40c 756c7793-9921-b854-b50c-673c26c4d40c}" +2024-07-05T13:37:16.734Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:16.62899215 +0000 UTC m=+102.624913310 (1 nodes), 51.508851ms from start of poll, 53.541147ms from cluster start. Timeout at 2024-07-05 13:37:16.734043607 +0000 UTC m=+102.729965059, 105.051749ms after stability +2024-07-05T13:37:16.734Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:16.839Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:16.734489557 +0000 UTC m=+102.730411009 (1 nodes), 6.125µs from start of poll, 159.038846ms from cluster start. Timeout at 2024-07-05 13:37:16.839517973 +0000 UTC m=+102.835438841, 105.027832ms after stability +2024-07-05T13:37:16.845Z [ERROR] server-6626ae65-698e-5ea0-4d01-c76e53837463: peer has newer term, stopping replication: peer="{Voter server-756c7793-9921-b854-b50c-673c26c4d40c 756c7793-9921-b854-b50c-673c26c4d40c}" +2024-07-05T13:37:16.845Z [INFO] server-6626ae65-698e-5ea0-4d01-c76e53837463: aborting pipeline replication: peer="{Voter server-756c7793-9921-b854-b50c-673c26c4d40c 756c7793-9921-b854-b50c-673c26c4d40c}" +2024-07-05T13:37:16.846Z [INFO] server-6626ae65-698e-5ea0-4d01-c76e53837463: entering follower state: follower="Node at 6626ae65-698e-5ea0-4d01-c76e53837463 [Follower]" leader-address= leader-id= +--- PASS: TestRaft_VerifyLeader_Fail (0.28s) === RUN TestRaft_VerifyLeader_PartialConnect -2024-07-05T13:31:00.138Z [INFO] server-5a05639c-67b6-0557-0c8c-536f04e2fa33: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a05639c-67b6-0557-0c8c-536f04e2fa33 Address:5a05639c-67b6-0557-0c8c-536f04e2fa33} {Suffrage:Voter ID:server-2a2b9df4-5b4f-3eaa-1521-951dd1b63cef Address:2a2b9df4-5b4f-3eaa-1521-951dd1b63cef} {Suffrage:Voter ID:server-94b94267-5e37-8330-63c2-cfe0c91de0af Address:94b94267-5e37-8330-63c2-cfe0c91de0af}]" -2024-07-05T13:31:00.138Z [INFO] server-2a2b9df4-5b4f-3eaa-1521-951dd1b63cef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a05639c-67b6-0557-0c8c-536f04e2fa33 Address:5a05639c-67b6-0557-0c8c-536f04e2fa33} {Suffrage:Voter ID:server-2a2b9df4-5b4f-3eaa-1521-951dd1b63cef Address:2a2b9df4-5b4f-3eaa-1521-951dd1b63cef} {Suffrage:Voter ID:server-94b94267-5e37-8330-63c2-cfe0c91de0af Address:94b94267-5e37-8330-63c2-cfe0c91de0af}]" -2024-07-05T13:31:00.138Z [INFO] server-94b94267-5e37-8330-63c2-cfe0c91de0af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a05639c-67b6-0557-0c8c-536f04e2fa33 Address:5a05639c-67b6-0557-0c8c-536f04e2fa33} {Suffrage:Voter ID:server-2a2b9df4-5b4f-3eaa-1521-951dd1b63cef Address:2a2b9df4-5b4f-3eaa-1521-951dd1b63cef} {Suffrage:Voter ID:server-94b94267-5e37-8330-63c2-cfe0c91de0af Address:94b94267-5e37-8330-63c2-cfe0c91de0af}]" -2024-07-05T13:31:00.138Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:00.138Z [INFO] server-5a05639c-67b6-0557-0c8c-536f04e2fa33: entering follower state: follower="Node at 5a05639c-67b6-0557-0c8c-536f04e2fa33 [Follower]" leader-address= leader-id= -2024-07-05T13:31:00.138Z [INFO] server-2a2b9df4-5b4f-3eaa-1521-951dd1b63cef: entering follower state: follower="Node at 2a2b9df4-5b4f-3eaa-1521-951dd1b63cef [Follower]" leader-address= leader-id= -2024-07-05T13:31:00.138Z [INFO] server-94b94267-5e37-8330-63c2-cfe0c91de0af: entering follower state: follower="Node at 94b94267-5e37-8330-63c2-cfe0c91de0af [Follower]" leader-address= leader-id= -2024-07-05T13:31:00.203Z [WARN] server-2a2b9df4-5b4f-3eaa-1521-951dd1b63cef: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:00.203Z [INFO] server-2a2b9df4-5b4f-3eaa-1521-951dd1b63cef: entering candidate state: node="Node at 2a2b9df4-5b4f-3eaa-1521-951dd1b63cef [Candidate]" term=2 -2024-07-05T13:31:00.204Z [INFO] server-2a2b9df4-5b4f-3eaa-1521-951dd1b63cef: election won: term=2 tally=2 -2024-07-05T13:31:00.204Z [INFO] server-2a2b9df4-5b4f-3eaa-1521-951dd1b63cef: entering leader state: leader="Node at 2a2b9df4-5b4f-3eaa-1521-951dd1b63cef [Leader]" -2024-07-05T13:31:00.204Z [INFO] server-2a2b9df4-5b4f-3eaa-1521-951dd1b63cef: added peer, starting replication: peer=server-5a05639c-67b6-0557-0c8c-536f04e2fa33 -2024-07-05T13:31:00.204Z [INFO] server-2a2b9df4-5b4f-3eaa-1521-951dd1b63cef: added peer, starting replication: peer=server-94b94267-5e37-8330-63c2-cfe0c91de0af -2024-07-05T13:31:00.204Z [INFO] server-2a2b9df4-5b4f-3eaa-1521-951dd1b63cef: pipelining replication: peer="{Voter server-5a05639c-67b6-0557-0c8c-536f04e2fa33 5a05639c-67b6-0557-0c8c-536f04e2fa33}" -2024-07-05T13:31:00.205Z [INFO] server-2a2b9df4-5b4f-3eaa-1521-951dd1b63cef: pipelining replication: peer="{Voter server-94b94267-5e37-8330-63c2-cfe0c91de0af 94b94267-5e37-8330-63c2-cfe0c91de0af}" -2024-07-05T13:31:00.310Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:00.204817374 +0000 UTC m=+100.974728049 (1 nodes), 66.030563ms from start of poll, 66.69006ms from cluster start. Timeout at 2024-07-05 13:31:00.310000044 +0000 UTC m=+101.079910719, 105.18267ms after stability -2024-07-05T13:31:00.315Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:31:00.421Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:31:00.315289498 +0000 UTC m=+101.085200173 (2 nodes), 2.48µs from start of poll, 177.162184ms from cluster start. Timeout at 2024-07-05 13:31:00.42156741 +0000 UTC m=+101.191478085, 106.277912ms after stability - raft_test.go:1968: [INFO] Disconnecting Node at 5a05639c-67b6-0557-0c8c-536f04e2fa33 [Follower] -2024-07-05T13:31:00.422Z [ERROR] server-2a2b9df4-5b4f-3eaa-1521-951dd1b63cef: failed to heartbeat to: peer=5a05639c-67b6-0557-0c8c-536f04e2fa33 backoff time=10ms error="failed to connect to peer: 5a05639c-67b6-0557-0c8c-536f04e2fa33" -2024-07-05T13:31:00.422Z [INFO] server-2a2b9df4-5b4f-3eaa-1521-951dd1b63cef: aborting pipeline replication: peer="{Voter server-94b94267-5e37-8330-63c2-cfe0c91de0af 94b94267-5e37-8330-63c2-cfe0c91de0af}" -2024-07-05T13:31:00.422Z [INFO] server-2a2b9df4-5b4f-3eaa-1521-951dd1b63cef: aborting pipeline replication: peer="{Voter server-5a05639c-67b6-0557-0c8c-536f04e2fa33 5a05639c-67b6-0557-0c8c-536f04e2fa33}" ---- PASS: TestRaft_VerifyLeader_PartialConnect (0.29s) +2024-07-05T13:37:16.855Z [INFO] server-4da25949-b0d0-5f22-f4b3-079ed886a8b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4da25949-b0d0-5f22-f4b3-079ed886a8b3 Address:4da25949-b0d0-5f22-f4b3-079ed886a8b3} {Suffrage:Voter ID:server-816c94ed-a58a-5343-d72c-ef5c1fc951aa Address:816c94ed-a58a-5343-d72c-ef5c1fc951aa} {Suffrage:Voter ID:server-ef0a1d86-7dca-ef0b-93f9-5ae2cc950bbc Address:ef0a1d86-7dca-ef0b-93f9-5ae2cc950bbc}]" +2024-07-05T13:37:16.856Z [INFO] server-816c94ed-a58a-5343-d72c-ef5c1fc951aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4da25949-b0d0-5f22-f4b3-079ed886a8b3 Address:4da25949-b0d0-5f22-f4b3-079ed886a8b3} {Suffrage:Voter ID:server-816c94ed-a58a-5343-d72c-ef5c1fc951aa Address:816c94ed-a58a-5343-d72c-ef5c1fc951aa} {Suffrage:Voter ID:server-ef0a1d86-7dca-ef0b-93f9-5ae2cc950bbc Address:ef0a1d86-7dca-ef0b-93f9-5ae2cc950bbc}]" +2024-07-05T13:37:16.856Z [INFO] server-ef0a1d86-7dca-ef0b-93f9-5ae2cc950bbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4da25949-b0d0-5f22-f4b3-079ed886a8b3 Address:4da25949-b0d0-5f22-f4b3-079ed886a8b3} {Suffrage:Voter ID:server-816c94ed-a58a-5343-d72c-ef5c1fc951aa Address:816c94ed-a58a-5343-d72c-ef5c1fc951aa} {Suffrage:Voter ID:server-ef0a1d86-7dca-ef0b-93f9-5ae2cc950bbc Address:ef0a1d86-7dca-ef0b-93f9-5ae2cc950bbc}]" +2024-07-05T13:37:16.856Z [INFO] server-4da25949-b0d0-5f22-f4b3-079ed886a8b3: entering follower state: follower="Node at 4da25949-b0d0-5f22-f4b3-079ed886a8b3 [Follower]" leader-address= leader-id= +2024-07-05T13:37:16.857Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:16.857Z [INFO] server-816c94ed-a58a-5343-d72c-ef5c1fc951aa: entering follower state: follower="Node at 816c94ed-a58a-5343-d72c-ef5c1fc951aa [Follower]" leader-address= leader-id= +2024-07-05T13:37:16.857Z [INFO] server-ef0a1d86-7dca-ef0b-93f9-5ae2cc950bbc: entering follower state: follower="Node at ef0a1d86-7dca-ef0b-93f9-5ae2cc950bbc [Follower]" leader-address= leader-id= +2024-07-05T13:37:16.954Z [WARN] server-4da25949-b0d0-5f22-f4b3-079ed886a8b3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:16.954Z [INFO] server-4da25949-b0d0-5f22-f4b3-079ed886a8b3: entering candidate state: node="Node at 4da25949-b0d0-5f22-f4b3-079ed886a8b3 [Candidate]" term=2 +2024-07-05T13:37:16.956Z [INFO] server-4da25949-b0d0-5f22-f4b3-079ed886a8b3: election won: term=2 tally=2 +2024-07-05T13:37:16.956Z [INFO] server-4da25949-b0d0-5f22-f4b3-079ed886a8b3: entering leader state: leader="Node at 4da25949-b0d0-5f22-f4b3-079ed886a8b3 [Leader]" +2024-07-05T13:37:16.956Z [INFO] server-4da25949-b0d0-5f22-f4b3-079ed886a8b3: added peer, starting replication: peer=server-816c94ed-a58a-5343-d72c-ef5c1fc951aa +2024-07-05T13:37:16.956Z [INFO] server-4da25949-b0d0-5f22-f4b3-079ed886a8b3: added peer, starting replication: peer=server-ef0a1d86-7dca-ef0b-93f9-5ae2cc950bbc +2024-07-05T13:37:16.958Z [INFO] server-4da25949-b0d0-5f22-f4b3-079ed886a8b3: pipelining replication: peer="{Voter server-816c94ed-a58a-5343-d72c-ef5c1fc951aa 816c94ed-a58a-5343-d72c-ef5c1fc951aa}" +2024-07-05T13:37:16.959Z [INFO] server-4da25949-b0d0-5f22-f4b3-079ed886a8b3: pipelining replication: peer="{Voter server-ef0a1d86-7dca-ef0b-93f9-5ae2cc950bbc ef0a1d86-7dca-ef0b-93f9-5ae2cc950bbc}" +2024-07-05T13:37:17.063Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:16.958181264 +0000 UTC m=+102.954102716 (1 nodes), 101.019822ms from start of poll, 103.152158ms from cluster start. Timeout at 2024-07-05 13:37:17.063721837 +0000 UTC m=+103.059642997, 105.540281ms after stability +2024-07-05T13:37:17.081Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:17.186Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:17.081612344 +0000 UTC m=+103.077533212 (2 nodes), 6.708µs from start of poll, 226.582654ms from cluster start. Timeout at 2024-07-05 13:37:17.186648342 +0000 UTC m=+103.182569502, 105.03629ms after stability + raft_test.go:1968: [INFO] Disconnecting Node at 816c94ed-a58a-5343-d72c-ef5c1fc951aa [Follower] +2024-07-05T13:37:17.187Z [ERROR] server-4da25949-b0d0-5f22-f4b3-079ed886a8b3: failed to heartbeat to: peer=816c94ed-a58a-5343-d72c-ef5c1fc951aa backoff time=10ms error="failed to connect to peer: 816c94ed-a58a-5343-d72c-ef5c1fc951aa" +2024-07-05T13:37:17.187Z [INFO] server-4da25949-b0d0-5f22-f4b3-079ed886a8b3: aborting pipeline replication: peer="{Voter server-ef0a1d86-7dca-ef0b-93f9-5ae2cc950bbc ef0a1d86-7dca-ef0b-93f9-5ae2cc950bbc}" +2024-07-05T13:37:17.187Z [INFO] server-4da25949-b0d0-5f22-f4b3-079ed886a8b3: aborting pipeline replication: peer="{Voter server-816c94ed-a58a-5343-d72c-ef5c1fc951aa 816c94ed-a58a-5343-d72c-ef5c1fc951aa}" +--- PASS: TestRaft_VerifyLeader_PartialConnect (0.34s) === RUN TestRaft_NotifyCh -2024-07-05T13:31:00.424Z [INFO] server-7f490d14-84b5-4745-5992-59e51262e950: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f490d14-84b5-4745-5992-59e51262e950 Address:7f490d14-84b5-4745-5992-59e51262e950}]" -2024-07-05T13:31:00.425Z [INFO] server-7f490d14-84b5-4745-5992-59e51262e950: entering follower state: follower="Node at 7f490d14-84b5-4745-5992-59e51262e950 [Follower]" leader-address= leader-id= -2024-07-05T13:31:00.509Z [WARN] server-7f490d14-84b5-4745-5992-59e51262e950: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:00.509Z [INFO] server-7f490d14-84b5-4745-5992-59e51262e950: entering candidate state: node="Node at 7f490d14-84b5-4745-5992-59e51262e950 [Candidate]" term=2 -2024-07-05T13:31:00.509Z [INFO] server-7f490d14-84b5-4745-5992-59e51262e950: election won: term=2 tally=1 -2024-07-05T13:31:00.509Z [INFO] server-7f490d14-84b5-4745-5992-59e51262e950: entering leader state: leader="Node at 7f490d14-84b5-4745-5992-59e51262e950 [Leader]" ---- PASS: TestRaft_NotifyCh (0.09s) +2024-07-05T13:37:17.195Z [INFO] server-7fff2ec0-eaf4-e73b-2b8c-6f0379073dd6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fff2ec0-eaf4-e73b-2b8c-6f0379073dd6 Address:7fff2ec0-eaf4-e73b-2b8c-6f0379073dd6}]" +2024-07-05T13:37:17.195Z [INFO] server-7fff2ec0-eaf4-e73b-2b8c-6f0379073dd6: entering follower state: follower="Node at 7fff2ec0-eaf4-e73b-2b8c-6f0379073dd6 [Follower]" leader-address= leader-id= +2024-07-05T13:37:17.250Z [WARN] server-7fff2ec0-eaf4-e73b-2b8c-6f0379073dd6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:17.250Z [INFO] server-7fff2ec0-eaf4-e73b-2b8c-6f0379073dd6: entering candidate state: node="Node at 7fff2ec0-eaf4-e73b-2b8c-6f0379073dd6 [Candidate]" term=2 +2024-07-05T13:37:17.250Z [INFO] server-7fff2ec0-eaf4-e73b-2b8c-6f0379073dd6: election won: term=2 tally=1 +2024-07-05T13:37:17.250Z [INFO] server-7fff2ec0-eaf4-e73b-2b8c-6f0379073dd6: entering leader state: leader="Node at 7fff2ec0-eaf4-e73b-2b8c-6f0379073dd6 [Leader]" +--- PASS: TestRaft_NotifyCh (0.06s) === RUN TestRaft_AppendEntry -2024-07-05T13:31:00.511Z [INFO] server-cc04868a-4b03-632e-1525-ac47fe558a16: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc04868a-4b03-632e-1525-ac47fe558a16 Address:cc04868a-4b03-632e-1525-ac47fe558a16} {Suffrage:Voter ID:server-e9323187-b6df-132f-0a41-620af6011c52 Address:e9323187-b6df-132f-0a41-620af6011c52} {Suffrage:Voter ID:server-3c08d2ef-fddb-a6a8-7b16-19d589a573eb Address:3c08d2ef-fddb-a6a8-7b16-19d589a573eb}]" -2024-07-05T13:31:00.511Z [INFO] server-e9323187-b6df-132f-0a41-620af6011c52: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc04868a-4b03-632e-1525-ac47fe558a16 Address:cc04868a-4b03-632e-1525-ac47fe558a16} {Suffrage:Voter ID:server-e9323187-b6df-132f-0a41-620af6011c52 Address:e9323187-b6df-132f-0a41-620af6011c52} {Suffrage:Voter ID:server-3c08d2ef-fddb-a6a8-7b16-19d589a573eb Address:3c08d2ef-fddb-a6a8-7b16-19d589a573eb}]" -2024-07-05T13:31:00.511Z [INFO] server-3c08d2ef-fddb-a6a8-7b16-19d589a573eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc04868a-4b03-632e-1525-ac47fe558a16 Address:cc04868a-4b03-632e-1525-ac47fe558a16} {Suffrage:Voter ID:server-e9323187-b6df-132f-0a41-620af6011c52 Address:e9323187-b6df-132f-0a41-620af6011c52} {Suffrage:Voter ID:server-3c08d2ef-fddb-a6a8-7b16-19d589a573eb Address:3c08d2ef-fddb-a6a8-7b16-19d589a573eb}]" -2024-07-05T13:31:00.512Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:31:00.512Z [INFO] server-cc04868a-4b03-632e-1525-ac47fe558a16: entering follower state: follower="Node at cc04868a-4b03-632e-1525-ac47fe558a16 [Follower]" leader-address= leader-id= -2024-07-05T13:31:00.513Z [INFO] server-e9323187-b6df-132f-0a41-620af6011c52: entering follower state: follower="Node at e9323187-b6df-132f-0a41-620af6011c52 [Follower]" leader-address= leader-id= -2024-07-05T13:31:00.513Z [INFO] server-3c08d2ef-fddb-a6a8-7b16-19d589a573eb: entering follower state: follower="Node at 3c08d2ef-fddb-a6a8-7b16-19d589a573eb [Follower]" leader-address= leader-id= -2024-07-05T13:31:00.577Z [WARN] server-3c08d2ef-fddb-a6a8-7b16-19d589a573eb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:00.578Z [INFO] server-3c08d2ef-fddb-a6a8-7b16-19d589a573eb: entering candidate state: node="Node at 3c08d2ef-fddb-a6a8-7b16-19d589a573eb [Candidate]" term=2 -2024-07-05T13:31:00.578Z [INFO] server-3c08d2ef-fddb-a6a8-7b16-19d589a573eb: election won: term=2 tally=2 -2024-07-05T13:31:00.578Z [INFO] server-3c08d2ef-fddb-a6a8-7b16-19d589a573eb: entering leader state: leader="Node at 3c08d2ef-fddb-a6a8-7b16-19d589a573eb [Leader]" -2024-07-05T13:31:00.578Z [INFO] server-3c08d2ef-fddb-a6a8-7b16-19d589a573eb: added peer, starting replication: peer=server-cc04868a-4b03-632e-1525-ac47fe558a16 -2024-07-05T13:31:00.578Z [INFO] server-3c08d2ef-fddb-a6a8-7b16-19d589a573eb: added peer, starting replication: peer=server-e9323187-b6df-132f-0a41-620af6011c52 -2024-07-05T13:31:00.578Z [INFO] server-3c08d2ef-fddb-a6a8-7b16-19d589a573eb: pipelining replication: peer="{Voter server-e9323187-b6df-132f-0a41-620af6011c52 e9323187-b6df-132f-0a41-620af6011c52}" -2024-07-05T13:31:00.578Z [INFO] server-3c08d2ef-fddb-a6a8-7b16-19d589a573eb: pipelining replication: peer="{Voter server-cc04868a-4b03-632e-1525-ac47fe558a16 cc04868a-4b03-632e-1525-ac47fe558a16}" -2024-07-05T13:31:00.684Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:31:00.579048924 +0000 UTC m=+101.348959599 (2 nodes), 67.001889ms from start of poll, 67.729823ms from cluster start. Timeout at 2024-07-05 13:31:00.684651219 +0000 UTC m=+101.454561894, 105.602295ms after stability -2024-07-05T13:31:00.684Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:00.790Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:00.684832453 +0000 UTC m=+101.454743128 (1 nodes), 1.8µs from start of poll, 173.513352ms from cluster start. Timeout at 2024-07-05 13:31:00.790075441 +0000 UTC m=+101.559986116, 105.242988ms after stability -2024-07-05T13:31:00.790Z [WARN] server-cc04868a-4b03-632e-1525-ac47fe558a16: clearing log suffix: from=1 to=2 -2024-07-05T13:31:00.790Z [WARN] server-cc04868a-4b03-632e-1525-ac47fe558a16: skipping application of old log: index=1 -2024-07-05T13:31:00.790Z [WARN] server-cc04868a-4b03-632e-1525-ac47fe558a16: skipping application of old log: index=1 -2024-07-05T13:31:00.790Z [INFO] server-3c08d2ef-fddb-a6a8-7b16-19d589a573eb: aborting pipeline replication: peer="{Voter server-cc04868a-4b03-632e-1525-ac47fe558a16 cc04868a-4b03-632e-1525-ac47fe558a16}" -2024-07-05T13:31:00.790Z [INFO] server-3c08d2ef-fddb-a6a8-7b16-19d589a573eb: aborting pipeline replication: peer="{Voter server-e9323187-b6df-132f-0a41-620af6011c52 e9323187-b6df-132f-0a41-620af6011c52}" ---- PASS: TestRaft_AppendEntry (0.28s) +2024-07-05T13:37:17.256Z [INFO] server-8087da58-9759-cd30-b3c5-013b0a8f66d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8087da58-9759-cd30-b3c5-013b0a8f66d5 Address:8087da58-9759-cd30-b3c5-013b0a8f66d5} {Suffrage:Voter ID:server-cb931bb1-b0ec-16e6-99ae-050056d398ba Address:cb931bb1-b0ec-16e6-99ae-050056d398ba} {Suffrage:Voter ID:server-cd9fc500-cae6-af0b-5172-9926eee10593 Address:cd9fc500-cae6-af0b-5172-9926eee10593}]" +2024-07-05T13:37:17.256Z [INFO] server-8087da58-9759-cd30-b3c5-013b0a8f66d5: entering follower state: follower="Node at 8087da58-9759-cd30-b3c5-013b0a8f66d5 [Follower]" leader-address= leader-id= +2024-07-05T13:37:17.257Z [INFO] server-cb931bb1-b0ec-16e6-99ae-050056d398ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8087da58-9759-cd30-b3c5-013b0a8f66d5 Address:8087da58-9759-cd30-b3c5-013b0a8f66d5} {Suffrage:Voter ID:server-cb931bb1-b0ec-16e6-99ae-050056d398ba Address:cb931bb1-b0ec-16e6-99ae-050056d398ba} {Suffrage:Voter ID:server-cd9fc500-cae6-af0b-5172-9926eee10593 Address:cd9fc500-cae6-af0b-5172-9926eee10593}]" +2024-07-05T13:37:17.257Z [INFO] server-cb931bb1-b0ec-16e6-99ae-050056d398ba: entering follower state: follower="Node at cb931bb1-b0ec-16e6-99ae-050056d398ba [Follower]" leader-address= leader-id= +2024-07-05T13:37:17.257Z [INFO] server-cd9fc500-cae6-af0b-5172-9926eee10593: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8087da58-9759-cd30-b3c5-013b0a8f66d5 Address:8087da58-9759-cd30-b3c5-013b0a8f66d5} {Suffrage:Voter ID:server-cb931bb1-b0ec-16e6-99ae-050056d398ba Address:cb931bb1-b0ec-16e6-99ae-050056d398ba} {Suffrage:Voter ID:server-cd9fc500-cae6-af0b-5172-9926eee10593 Address:cd9fc500-cae6-af0b-5172-9926eee10593}]" +2024-07-05T13:37:17.257Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:17.257Z [INFO] server-cd9fc500-cae6-af0b-5172-9926eee10593: entering follower state: follower="Node at cd9fc500-cae6-af0b-5172-9926eee10593 [Follower]" leader-address= leader-id= +2024-07-05T13:37:17.335Z [WARN] server-cd9fc500-cae6-af0b-5172-9926eee10593: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:17.335Z [INFO] server-cd9fc500-cae6-af0b-5172-9926eee10593: entering candidate state: node="Node at cd9fc500-cae6-af0b-5172-9926eee10593 [Candidate]" term=2 +2024-07-05T13:37:17.335Z [INFO] server-cd9fc500-cae6-af0b-5172-9926eee10593: election won: term=2 tally=2 +2024-07-05T13:37:17.336Z [INFO] server-cd9fc500-cae6-af0b-5172-9926eee10593: entering leader state: leader="Node at cd9fc500-cae6-af0b-5172-9926eee10593 [Leader]" +2024-07-05T13:37:17.336Z [INFO] server-cd9fc500-cae6-af0b-5172-9926eee10593: added peer, starting replication: peer=server-8087da58-9759-cd30-b3c5-013b0a8f66d5 +2024-07-05T13:37:17.336Z [INFO] server-cd9fc500-cae6-af0b-5172-9926eee10593: added peer, starting replication: peer=server-cb931bb1-b0ec-16e6-99ae-050056d398ba +2024-07-05T13:37:17.337Z [INFO] server-cd9fc500-cae6-af0b-5172-9926eee10593: pipelining replication: peer="{Voter server-8087da58-9759-cd30-b3c5-013b0a8f66d5 8087da58-9759-cd30-b3c5-013b0a8f66d5}" +2024-07-05T13:37:17.337Z [INFO] server-cd9fc500-cae6-af0b-5172-9926eee10593: pipelining replication: peer="{Voter server-cb931bb1-b0ec-16e6-99ae-050056d398ba cb931bb1-b0ec-16e6-99ae-050056d398ba}" +2024-07-05T13:37:17.442Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:17.337103303 +0000 UTC m=+103.333024463 (2 nodes), 79.11868ms from start of poll, 80.94098ms from cluster start. Timeout at 2024-07-05 13:37:17.442246925 +0000 UTC m=+103.438167793, 105.14333ms after stability +2024-07-05T13:37:17.442Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:17.548Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:17.442563086 +0000 UTC m=+103.438484246 (1 nodes), 4.084µs from start of poll, 186.400763ms from cluster start. Timeout at 2024-07-05 13:37:17.548638857 +0000 UTC m=+103.544560309, 106.076063ms after stability +2024-07-05T13:37:17.549Z [WARN] server-8087da58-9759-cd30-b3c5-013b0a8f66d5: clearing log suffix: from=1 to=2 +2024-07-05T13:37:17.549Z [WARN] server-8087da58-9759-cd30-b3c5-013b0a8f66d5: skipping application of old log: index=1 +2024-07-05T13:37:17.549Z [WARN] server-8087da58-9759-cd30-b3c5-013b0a8f66d5: skipping application of old log: index=1 +2024-07-05T13:37:17.551Z [INFO] server-cd9fc500-cae6-af0b-5172-9926eee10593: aborting pipeline replication: peer="{Voter server-8087da58-9759-cd30-b3c5-013b0a8f66d5 8087da58-9759-cd30-b3c5-013b0a8f66d5}" +2024-07-05T13:37:17.551Z [INFO] server-cd9fc500-cae6-af0b-5172-9926eee10593: aborting pipeline replication: peer="{Voter server-cb931bb1-b0ec-16e6-99ae-050056d398ba cb931bb1-b0ec-16e6-99ae-050056d398ba}" +--- PASS: TestRaft_AppendEntry (0.30s) === RUN TestRaft_VotingGrant_WhenLeaderAvailable -2024-07-05T13:31:00.792Z [INFO] server-46f63d5f-d384-87cf-37b9-7599404a8343: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46f63d5f-d384-87cf-37b9-7599404a8343 Address:46f63d5f-d384-87cf-37b9-7599404a8343} {Suffrage:Voter ID:server-36084133-8d95-b39f-78f9-0a93430423db Address:36084133-8d95-b39f-78f9-0a93430423db} {Suffrage:Voter ID:server-e69e932e-a201-0357-140f-36f6a2c57e97 Address:e69e932e-a201-0357-140f-36f6a2c57e97}]" -2024-07-05T13:31:00.793Z [INFO] server-36084133-8d95-b39f-78f9-0a93430423db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46f63d5f-d384-87cf-37b9-7599404a8343 Address:46f63d5f-d384-87cf-37b9-7599404a8343} {Suffrage:Voter ID:server-36084133-8d95-b39f-78f9-0a93430423db Address:36084133-8d95-b39f-78f9-0a93430423db} {Suffrage:Voter ID:server-e69e932e-a201-0357-140f-36f6a2c57e97 Address:e69e932e-a201-0357-140f-36f6a2c57e97}]" -2024-07-05T13:31:00.793Z [INFO] server-e69e932e-a201-0357-140f-36f6a2c57e97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46f63d5f-d384-87cf-37b9-7599404a8343 Address:46f63d5f-d384-87cf-37b9-7599404a8343} {Suffrage:Voter ID:server-36084133-8d95-b39f-78f9-0a93430423db Address:36084133-8d95-b39f-78f9-0a93430423db} {Suffrage:Voter ID:server-e69e932e-a201-0357-140f-36f6a2c57e97 Address:e69e932e-a201-0357-140f-36f6a2c57e97}]" -2024-07-05T13:31:00.793Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:31:00.793Z [INFO] server-46f63d5f-d384-87cf-37b9-7599404a8343: entering follower state: follower="Node at 46f63d5f-d384-87cf-37b9-7599404a8343 [Follower]" leader-address= leader-id= -2024-07-05T13:31:00.793Z [INFO] server-36084133-8d95-b39f-78f9-0a93430423db: entering follower state: follower="Node at 36084133-8d95-b39f-78f9-0a93430423db [Follower]" leader-address= leader-id= -2024-07-05T13:31:00.793Z [INFO] server-e69e932e-a201-0357-140f-36f6a2c57e97: entering follower state: follower="Node at e69e932e-a201-0357-140f-36f6a2c57e97 [Follower]" leader-address= leader-id= -2024-07-05T13:31:00.845Z [WARN] server-36084133-8d95-b39f-78f9-0a93430423db: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:00.845Z [INFO] server-36084133-8d95-b39f-78f9-0a93430423db: entering candidate state: node="Node at 36084133-8d95-b39f-78f9-0a93430423db [Candidate]" term=2 -2024-07-05T13:31:00.845Z [INFO] server-36084133-8d95-b39f-78f9-0a93430423db: election won: term=2 tally=2 -2024-07-05T13:31:00.845Z [INFO] server-36084133-8d95-b39f-78f9-0a93430423db: entering leader state: leader="Node at 36084133-8d95-b39f-78f9-0a93430423db [Leader]" -2024-07-05T13:31:00.845Z [INFO] server-36084133-8d95-b39f-78f9-0a93430423db: added peer, starting replication: peer=server-46f63d5f-d384-87cf-37b9-7599404a8343 -2024-07-05T13:31:00.846Z [INFO] server-36084133-8d95-b39f-78f9-0a93430423db: added peer, starting replication: peer=server-e69e932e-a201-0357-140f-36f6a2c57e97 -2024-07-05T13:31:00.846Z [INFO] server-36084133-8d95-b39f-78f9-0a93430423db: pipelining replication: peer="{Voter server-e69e932e-a201-0357-140f-36f6a2c57e97 e69e932e-a201-0357-140f-36f6a2c57e97}" -2024-07-05T13:31:00.854Z [INFO] server-36084133-8d95-b39f-78f9-0a93430423db: pipelining replication: peer="{Voter server-46f63d5f-d384-87cf-37b9-7599404a8343 46f63d5f-d384-87cf-37b9-7599404a8343}" -2024-07-05T13:31:00.959Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:31:00.853783925 +0000 UTC m=+101.623694600 (2 nodes), 60.383601ms from start of poll, 61.036178ms from cluster start. Timeout at 2024-07-05 13:31:00.959520416 +0000 UTC m=+101.729431091, 105.736491ms after stability -2024-07-05T13:31:00.959Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:01.064Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:00.95969533 +0000 UTC m=+101.729606005 (1 nodes), 1.96µs from start of poll, 166.947583ms from cluster start. Timeout at 2024-07-05 13:31:01.064721645 +0000 UTC m=+101.834632320, 105.026315ms after stability -2024-07-05T13:31:01.064Z [WARN] server-e69e932e-a201-0357-140f-36f6a2c57e97: rejecting vote request since we have a leader: from=46f63d5f-d384-87cf-37b9-7599404a8343 leader=36084133-8d95-b39f-78f9-0a93430423db leader-id=server-36084133-8d95-b39f-78f9-0a93430423db -2024-07-05T13:31:01.065Z [ERROR] server-36084133-8d95-b39f-78f9-0a93430423db: peer has newer term, stopping replication: peer="{Voter server-e69e932e-a201-0357-140f-36f6a2c57e97 e69e932e-a201-0357-140f-36f6a2c57e97}" -2024-07-05T13:31:01.065Z [INFO] server-36084133-8d95-b39f-78f9-0a93430423db: aborting pipeline replication: peer="{Voter server-e69e932e-a201-0357-140f-36f6a2c57e97 e69e932e-a201-0357-140f-36f6a2c57e97}" -2024-07-05T13:31:01.065Z [INFO] server-36084133-8d95-b39f-78f9-0a93430423db: aborting pipeline replication: peer="{Voter server-46f63d5f-d384-87cf-37b9-7599404a8343 46f63d5f-d384-87cf-37b9-7599404a8343}" +2024-07-05T13:37:17.561Z [INFO] server-73350296-35e6-5bf5-f491-eb684ba93492: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73350296-35e6-5bf5-f491-eb684ba93492 Address:73350296-35e6-5bf5-f491-eb684ba93492} {Suffrage:Voter ID:server-4645715e-81d5-1f32-fffd-5b2d49e10897 Address:4645715e-81d5-1f32-fffd-5b2d49e10897} {Suffrage:Voter ID:server-2fcb22e1-0b4b-8593-3f80-43a7f95c3774 Address:2fcb22e1-0b4b-8593-3f80-43a7f95c3774}]" +2024-07-05T13:37:17.562Z [INFO] server-73350296-35e6-5bf5-f491-eb684ba93492: entering follower state: follower="Node at 73350296-35e6-5bf5-f491-eb684ba93492 [Follower]" leader-address= leader-id= +2024-07-05T13:37:17.562Z [INFO] server-4645715e-81d5-1f32-fffd-5b2d49e10897: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73350296-35e6-5bf5-f491-eb684ba93492 Address:73350296-35e6-5bf5-f491-eb684ba93492} {Suffrage:Voter ID:server-4645715e-81d5-1f32-fffd-5b2d49e10897 Address:4645715e-81d5-1f32-fffd-5b2d49e10897} {Suffrage:Voter ID:server-2fcb22e1-0b4b-8593-3f80-43a7f95c3774 Address:2fcb22e1-0b4b-8593-3f80-43a7f95c3774}]" +2024-07-05T13:37:17.562Z [INFO] server-4645715e-81d5-1f32-fffd-5b2d49e10897: entering follower state: follower="Node at 4645715e-81d5-1f32-fffd-5b2d49e10897 [Follower]" leader-address= leader-id= +2024-07-05T13:37:17.562Z [INFO] server-2fcb22e1-0b4b-8593-3f80-43a7f95c3774: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73350296-35e6-5bf5-f491-eb684ba93492 Address:73350296-35e6-5bf5-f491-eb684ba93492} {Suffrage:Voter ID:server-4645715e-81d5-1f32-fffd-5b2d49e10897 Address:4645715e-81d5-1f32-fffd-5b2d49e10897} {Suffrage:Voter ID:server-2fcb22e1-0b4b-8593-3f80-43a7f95c3774 Address:2fcb22e1-0b4b-8593-3f80-43a7f95c3774}]" +2024-07-05T13:37:17.563Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:17.563Z [INFO] server-2fcb22e1-0b4b-8593-3f80-43a7f95c3774: entering follower state: follower="Node at 2fcb22e1-0b4b-8593-3f80-43a7f95c3774 [Follower]" leader-address= leader-id= +2024-07-05T13:37:17.623Z [WARN] server-4645715e-81d5-1f32-fffd-5b2d49e10897: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:17.623Z [INFO] server-4645715e-81d5-1f32-fffd-5b2d49e10897: entering candidate state: node="Node at 4645715e-81d5-1f32-fffd-5b2d49e10897 [Candidate]" term=2 +2024-07-05T13:37:17.624Z [INFO] server-4645715e-81d5-1f32-fffd-5b2d49e10897: election won: term=2 tally=2 +2024-07-05T13:37:17.624Z [INFO] server-4645715e-81d5-1f32-fffd-5b2d49e10897: entering leader state: leader="Node at 4645715e-81d5-1f32-fffd-5b2d49e10897 [Leader]" +2024-07-05T13:37:17.624Z [INFO] server-4645715e-81d5-1f32-fffd-5b2d49e10897: added peer, starting replication: peer=server-73350296-35e6-5bf5-f491-eb684ba93492 +2024-07-05T13:37:17.624Z [INFO] server-4645715e-81d5-1f32-fffd-5b2d49e10897: added peer, starting replication: peer=server-2fcb22e1-0b4b-8593-3f80-43a7f95c3774 +2024-07-05T13:37:17.625Z [INFO] server-4645715e-81d5-1f32-fffd-5b2d49e10897: pipelining replication: peer="{Voter server-73350296-35e6-5bf5-f491-eb684ba93492 73350296-35e6-5bf5-f491-eb684ba93492}" +2024-07-05T13:37:17.625Z [INFO] server-4645715e-81d5-1f32-fffd-5b2d49e10897: pipelining replication: peer="{Voter server-2fcb22e1-0b4b-8593-3f80-43a7f95c3774 2fcb22e1-0b4b-8593-3f80-43a7f95c3774}" +2024-07-05T13:37:17.729Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:17.624258184 +0000 UTC m=+103.620179636 (2 nodes), 61.131634ms from start of poll, 63.166263ms from cluster start. Timeout at 2024-07-05 13:37:17.729795549 +0000 UTC m=+103.725716417, 105.536781ms after stability +2024-07-05T13:37:17.730Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:17.835Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:17.73008546 +0000 UTC m=+103.726006329 (1 nodes), 3.792µs from start of poll, 168.992956ms from cluster start. Timeout at 2024-07-05 13:37:17.835250082 +0000 UTC m=+103.831171241, 105.164912ms after stability +2024-07-05T13:37:17.835Z [WARN] server-2fcb22e1-0b4b-8593-3f80-43a7f95c3774: rejecting vote request since we have a leader: from=73350296-35e6-5bf5-f491-eb684ba93492 leader=4645715e-81d5-1f32-fffd-5b2d49e10897 leader-id=server-4645715e-81d5-1f32-fffd-5b2d49e10897 +2024-07-05T13:37:17.836Z [INFO] server-4645715e-81d5-1f32-fffd-5b2d49e10897: aborting pipeline replication: peer="{Voter server-2fcb22e1-0b4b-8593-3f80-43a7f95c3774 2fcb22e1-0b4b-8593-3f80-43a7f95c3774}" +2024-07-05T13:37:17.836Z [INFO] server-4645715e-81d5-1f32-fffd-5b2d49e10897: aborting pipeline replication: peer="{Voter server-73350296-35e6-5bf5-f491-eb684ba93492 73350296-35e6-5bf5-f491-eb684ba93492}" --- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.28s) === RUN TestRaft_ProtocolVersion_RejectRPC -2024-07-05T13:31:01.068Z [INFO] server-36d37649-901a-5506-c61c-ac1a92587d91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36d37649-901a-5506-c61c-ac1a92587d91 Address:36d37649-901a-5506-c61c-ac1a92587d91} {Suffrage:Voter ID:server-303b6c72-6ab5-d6f6-5711-fd6403e8a2dd Address:303b6c72-6ab5-d6f6-5711-fd6403e8a2dd} {Suffrage:Voter ID:server-fbada287-9315-d2a3-aefa-90c7309edc52 Address:fbada287-9315-d2a3-aefa-90c7309edc52}]" -2024-07-05T13:31:01.069Z [INFO] server-303b6c72-6ab5-d6f6-5711-fd6403e8a2dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36d37649-901a-5506-c61c-ac1a92587d91 Address:36d37649-901a-5506-c61c-ac1a92587d91} {Suffrage:Voter ID:server-303b6c72-6ab5-d6f6-5711-fd6403e8a2dd Address:303b6c72-6ab5-d6f6-5711-fd6403e8a2dd} {Suffrage:Voter ID:server-fbada287-9315-d2a3-aefa-90c7309edc52 Address:fbada287-9315-d2a3-aefa-90c7309edc52}]" -2024-07-05T13:31:01.069Z [INFO] server-fbada287-9315-d2a3-aefa-90c7309edc52: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36d37649-901a-5506-c61c-ac1a92587d91 Address:36d37649-901a-5506-c61c-ac1a92587d91} {Suffrage:Voter ID:server-303b6c72-6ab5-d6f6-5711-fd6403e8a2dd Address:303b6c72-6ab5-d6f6-5711-fd6403e8a2dd} {Suffrage:Voter ID:server-fbada287-9315-d2a3-aefa-90c7309edc52 Address:fbada287-9315-d2a3-aefa-90c7309edc52}]" -2024-07-05T13:31:01.069Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:31:01.069Z [INFO] server-36d37649-901a-5506-c61c-ac1a92587d91: entering follower state: follower="Node at 36d37649-901a-5506-c61c-ac1a92587d91 [Follower]" leader-address= leader-id= -2024-07-05T13:31:01.069Z [INFO] server-303b6c72-6ab5-d6f6-5711-fd6403e8a2dd: entering follower state: follower="Node at 303b6c72-6ab5-d6f6-5711-fd6403e8a2dd [Follower]" leader-address= leader-id= -2024-07-05T13:31:01.069Z [INFO] server-fbada287-9315-d2a3-aefa-90c7309edc52: entering follower state: follower="Node at fbada287-9315-d2a3-aefa-90c7309edc52 [Follower]" leader-address= leader-id= -2024-07-05T13:31:01.137Z [WARN] server-303b6c72-6ab5-d6f6-5711-fd6403e8a2dd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:01.137Z [INFO] server-303b6c72-6ab5-d6f6-5711-fd6403e8a2dd: entering candidate state: node="Node at 303b6c72-6ab5-d6f6-5711-fd6403e8a2dd [Candidate]" term=2 -2024-07-05T13:31:01.137Z [INFO] server-303b6c72-6ab5-d6f6-5711-fd6403e8a2dd: election won: term=2 tally=2 -2024-07-05T13:31:01.137Z [INFO] server-303b6c72-6ab5-d6f6-5711-fd6403e8a2dd: entering leader state: leader="Node at 303b6c72-6ab5-d6f6-5711-fd6403e8a2dd [Leader]" -2024-07-05T13:31:01.137Z [INFO] server-303b6c72-6ab5-d6f6-5711-fd6403e8a2dd: added peer, starting replication: peer=server-36d37649-901a-5506-c61c-ac1a92587d91 -2024-07-05T13:31:01.137Z [INFO] server-303b6c72-6ab5-d6f6-5711-fd6403e8a2dd: added peer, starting replication: peer=server-fbada287-9315-d2a3-aefa-90c7309edc52 -2024-07-05T13:31:01.137Z [INFO] server-303b6c72-6ab5-d6f6-5711-fd6403e8a2dd: pipelining replication: peer="{Voter server-fbada287-9315-d2a3-aefa-90c7309edc52 fbada287-9315-d2a3-aefa-90c7309edc52}" -2024-07-05T13:31:01.138Z [WARN] server-fbada287-9315-d2a3-aefa-90c7309edc52: rejecting vote request since our last term is greater: candidate=303b6c72-6ab5-d6f6-5711-fd6403e8a2dd last-term=2 last-candidate-term=1 -2024-07-05T13:31:01.138Z [INFO] server-303b6c72-6ab5-d6f6-5711-fd6403e8a2dd: pipelining replication: peer="{Voter server-36d37649-901a-5506-c61c-ac1a92587d91 36d37649-901a-5506-c61c-ac1a92587d91}" -2024-07-05T13:31:01.248Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:31:01.138315423 +0000 UTC m=+101.908226098 (2 nodes), 68.967381ms from start of poll, 69.650117ms from cluster start. Timeout at 2024-07-05 13:31:01.244554975 +0000 UTC m=+102.014465650, 106.239552ms after stability -2024-07-05T13:31:01.250Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:01.355Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:01.25012066 +0000 UTC m=+102.020031335 (1 nodes), 1.6µs from start of poll, 181.455354ms from cluster start. Timeout at 2024-07-05 13:31:01.355549361 +0000 UTC m=+102.125460036, 105.428701ms after stability -2024-07-05T13:31:01.356Z [INFO] server-303b6c72-6ab5-d6f6-5711-fd6403e8a2dd: aborting pipeline replication: peer="{Voter server-fbada287-9315-d2a3-aefa-90c7309edc52 fbada287-9315-d2a3-aefa-90c7309edc52}" -2024-07-05T13:31:01.356Z [INFO] server-303b6c72-6ab5-d6f6-5711-fd6403e8a2dd: aborting pipeline replication: peer="{Voter server-36d37649-901a-5506-c61c-ac1a92587d91 36d37649-901a-5506-c61c-ac1a92587d91}" ---- PASS: TestRaft_ProtocolVersion_RejectRPC (0.29s) +2024-07-05T13:37:17.855Z [INFO] server-2244eb32-5315-3e15-878a-433dcecb1053: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2244eb32-5315-3e15-878a-433dcecb1053 Address:2244eb32-5315-3e15-878a-433dcecb1053} {Suffrage:Voter ID:server-c2664264-fb29-1a19-302e-357ba26325be Address:c2664264-fb29-1a19-302e-357ba26325be} {Suffrage:Voter ID:server-21698169-bfe2-7876-a66c-f4313baee501 Address:21698169-bfe2-7876-a66c-f4313baee501}]" +2024-07-05T13:37:17.856Z [INFO] server-c2664264-fb29-1a19-302e-357ba26325be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2244eb32-5315-3e15-878a-433dcecb1053 Address:2244eb32-5315-3e15-878a-433dcecb1053} {Suffrage:Voter ID:server-c2664264-fb29-1a19-302e-357ba26325be Address:c2664264-fb29-1a19-302e-357ba26325be} {Suffrage:Voter ID:server-21698169-bfe2-7876-a66c-f4313baee501 Address:21698169-bfe2-7876-a66c-f4313baee501}]" +2024-07-05T13:37:17.858Z [INFO] server-2244eb32-5315-3e15-878a-433dcecb1053: entering follower state: follower="Node at 2244eb32-5315-3e15-878a-433dcecb1053 [Follower]" leader-address= leader-id= +2024-07-05T13:37:17.857Z [INFO] server-21698169-bfe2-7876-a66c-f4313baee501: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2244eb32-5315-3e15-878a-433dcecb1053 Address:2244eb32-5315-3e15-878a-433dcecb1053} {Suffrage:Voter ID:server-c2664264-fb29-1a19-302e-357ba26325be Address:c2664264-fb29-1a19-302e-357ba26325be} {Suffrage:Voter ID:server-21698169-bfe2-7876-a66c-f4313baee501 Address:21698169-bfe2-7876-a66c-f4313baee501}]" +2024-07-05T13:37:17.859Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:17.860Z [INFO] server-21698169-bfe2-7876-a66c-f4313baee501: entering follower state: follower="Node at 21698169-bfe2-7876-a66c-f4313baee501 [Follower]" leader-address= leader-id= +2024-07-05T13:37:17.861Z [INFO] server-c2664264-fb29-1a19-302e-357ba26325be: entering follower state: follower="Node at c2664264-fb29-1a19-302e-357ba26325be [Follower]" leader-address= leader-id= +2024-07-05T13:37:17.909Z [WARN] server-2244eb32-5315-3e15-878a-433dcecb1053: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:17.909Z [INFO] server-2244eb32-5315-3e15-878a-433dcecb1053: entering candidate state: node="Node at 2244eb32-5315-3e15-878a-433dcecb1053 [Candidate]" term=2 +2024-07-05T13:37:17.909Z [INFO] server-2244eb32-5315-3e15-878a-433dcecb1053: election won: term=2 tally=2 +2024-07-05T13:37:17.910Z [INFO] server-2244eb32-5315-3e15-878a-433dcecb1053: entering leader state: leader="Node at 2244eb32-5315-3e15-878a-433dcecb1053 [Leader]" +2024-07-05T13:37:17.910Z [INFO] server-2244eb32-5315-3e15-878a-433dcecb1053: added peer, starting replication: peer=server-c2664264-fb29-1a19-302e-357ba26325be +2024-07-05T13:37:17.910Z [INFO] server-2244eb32-5315-3e15-878a-433dcecb1053: added peer, starting replication: peer=server-21698169-bfe2-7876-a66c-f4313baee501 +2024-07-05T13:37:17.910Z [INFO] server-2244eb32-5315-3e15-878a-433dcecb1053: pipelining replication: peer="{Voter server-c2664264-fb29-1a19-302e-357ba26325be c2664264-fb29-1a19-302e-357ba26325be}" +2024-07-05T13:37:17.911Z [INFO] server-2244eb32-5315-3e15-878a-433dcecb1053: pipelining replication: peer="{Voter server-21698169-bfe2-7876-a66c-f4313baee501 21698169-bfe2-7876-a66c-f4313baee501}" +2024-07-05T13:37:18.016Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:17.910130047 +0000 UTC m=+103.906050916 (2 nodes), 50.425913ms from start of poll, 55.14966ms from cluster start. Timeout at 2024-07-05 13:37:18.015809451 +0000 UTC m=+104.011730319, 105.679403ms after stability +2024-07-05T13:37:18.016Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:18.121Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:18.016328317 +0000 UTC m=+104.012249185 (1 nodes), 4.667µs from start of poll, 161.347929ms from cluster start. Timeout at 2024-07-05 13:37:18.121357607 +0000 UTC m=+104.117278767, 105.029582ms after stability +2024-07-05T13:37:18.121Z [INFO] server-2244eb32-5315-3e15-878a-433dcecb1053: aborting pipeline replication: peer="{Voter server-c2664264-fb29-1a19-302e-357ba26325be c2664264-fb29-1a19-302e-357ba26325be}" +2024-07-05T13:37:18.121Z [INFO] server-2244eb32-5315-3e15-878a-433dcecb1053: aborting pipeline replication: peer="{Voter server-21698169-bfe2-7876-a66c-f4313baee501 21698169-bfe2-7876-a66c-f4313baee501}" +--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.28s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 raft_test.go:2151: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s) @@ -3534,271 +3590,274 @@ raft_test.go:2196: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s) === RUN TestRaft_LeaderID_Propagated -2024-07-05T13:31:01.359Z [INFO] server-1ef6ebdc-853c-1286-7841-57873c9eeb82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ef6ebdc-853c-1286-7841-57873c9eeb82 Address:1ef6ebdc-853c-1286-7841-57873c9eeb82} {Suffrage:Voter ID:server-d5295a1b-2aa3-930c-ec34-7acb502246af Address:d5295a1b-2aa3-930c-ec34-7acb502246af} {Suffrage:Voter ID:server-966878d3-215d-9e09-1dee-0e44387e7554 Address:966878d3-215d-9e09-1dee-0e44387e7554}]" -2024-07-05T13:31:01.359Z [INFO] server-1ef6ebdc-853c-1286-7841-57873c9eeb82: entering follower state: follower="Node at 1ef6ebdc-853c-1286-7841-57873c9eeb82 [Follower]" leader-address= leader-id= -2024-07-05T13:31:01.359Z [INFO] server-d5295a1b-2aa3-930c-ec34-7acb502246af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ef6ebdc-853c-1286-7841-57873c9eeb82 Address:1ef6ebdc-853c-1286-7841-57873c9eeb82} {Suffrage:Voter ID:server-d5295a1b-2aa3-930c-ec34-7acb502246af Address:d5295a1b-2aa3-930c-ec34-7acb502246af} {Suffrage:Voter ID:server-966878d3-215d-9e09-1dee-0e44387e7554 Address:966878d3-215d-9e09-1dee-0e44387e7554}]" -2024-07-05T13:31:01.359Z [INFO] server-d5295a1b-2aa3-930c-ec34-7acb502246af: entering follower state: follower="Node at d5295a1b-2aa3-930c-ec34-7acb502246af [Follower]" leader-address= leader-id= -2024-07-05T13:31:01.359Z [INFO] server-966878d3-215d-9e09-1dee-0e44387e7554: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ef6ebdc-853c-1286-7841-57873c9eeb82 Address:1ef6ebdc-853c-1286-7841-57873c9eeb82} {Suffrage:Voter ID:server-d5295a1b-2aa3-930c-ec34-7acb502246af Address:d5295a1b-2aa3-930c-ec34-7acb502246af} {Suffrage:Voter ID:server-966878d3-215d-9e09-1dee-0e44387e7554 Address:966878d3-215d-9e09-1dee-0e44387e7554}]" -2024-07-05T13:31:01.359Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:01.359Z [INFO] server-966878d3-215d-9e09-1dee-0e44387e7554: entering follower state: follower="Node at 966878d3-215d-9e09-1dee-0e44387e7554 [Follower]" leader-address= leader-id= -2024-07-05T13:31:01.432Z [WARN] server-1ef6ebdc-853c-1286-7841-57873c9eeb82: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:01.432Z [INFO] server-1ef6ebdc-853c-1286-7841-57873c9eeb82: entering candidate state: node="Node at 1ef6ebdc-853c-1286-7841-57873c9eeb82 [Candidate]" term=2 -2024-07-05T13:31:01.432Z [INFO] server-1ef6ebdc-853c-1286-7841-57873c9eeb82: election won: term=2 tally=2 -2024-07-05T13:31:01.433Z [INFO] server-1ef6ebdc-853c-1286-7841-57873c9eeb82: entering leader state: leader="Node at 1ef6ebdc-853c-1286-7841-57873c9eeb82 [Leader]" -2024-07-05T13:31:01.433Z [INFO] server-1ef6ebdc-853c-1286-7841-57873c9eeb82: added peer, starting replication: peer=server-d5295a1b-2aa3-930c-ec34-7acb502246af -2024-07-05T13:31:01.433Z [INFO] server-1ef6ebdc-853c-1286-7841-57873c9eeb82: added peer, starting replication: peer=server-966878d3-215d-9e09-1dee-0e44387e7554 -2024-07-05T13:31:01.433Z [INFO] server-1ef6ebdc-853c-1286-7841-57873c9eeb82: pipelining replication: peer="{Voter server-966878d3-215d-9e09-1dee-0e44387e7554 966878d3-215d-9e09-1dee-0e44387e7554}" -2024-07-05T13:31:01.433Z [INFO] server-1ef6ebdc-853c-1286-7841-57873c9eeb82: pipelining replication: peer="{Voter server-d5295a1b-2aa3-930c-ec34-7acb502246af d5295a1b-2aa3-930c-ec34-7acb502246af}" -2024-07-05T13:31:01.538Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:01.433705699 +0000 UTC m=+102.203616374 (1 nodes), 73.82681ms from start of poll, 74.559664ms from cluster start. Timeout at 2024-07-05 13:31:01.538788852 +0000 UTC m=+102.308699527, 105.083153ms after stability -2024-07-05T13:31:01.538Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:01.645Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:01.539004885 +0000 UTC m=+102.308915560 (1 nodes), 1.36µs from start of poll, 179.85885ms from cluster start. Timeout at 2024-07-05 13:31:01.645223998 +0000 UTC m=+102.415134673, 106.219113ms after stability -2024-07-05T13:31:01.648Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:01.652Z [WARN] server-966878d3-215d-9e09-1dee-0e44387e7554: skipping application of old log: index=2 -2024-07-05T13:31:01.756Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:01.648236732 +0000 UTC m=+102.418147407 (1 nodes), 2.4µs from start of poll, 289.090697ms from cluster start. Timeout at 2024-07-05 13:31:01.756428257 +0000 UTC m=+102.526338932, 108.191525ms after stability -2024-07-05T13:31:01.756Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:01.864Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:01.756878561 +0000 UTC m=+102.526789236 (1 nodes), 1.44µs from start of poll, 397.732526ms from cluster start. Timeout at 2024-07-05 13:31:01.864548983 +0000 UTC m=+102.634459658, 107.670422ms after stability -2024-07-05T13:31:01.865Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:01.970Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:01.865518189 +0000 UTC m=+102.635428864 (1 nodes), 1.56µs from start of poll, 506.372154ms from cluster start. Timeout at 2024-07-05 13:31:01.970748537 +0000 UTC m=+102.740659212, 105.230348ms after stability -2024-07-05T13:31:01.971Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:02.077Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:01.971322157 +0000 UTC m=+102.741232832 (1 nodes), 1.64µs from start of poll, 612.176122ms from cluster start. Timeout at 2024-07-05 13:31:02.077101246 +0000 UTC m=+102.847012001, 105.779169ms after stability -2024-07-05T13:31:02.182Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:02.288Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:02.18273878 +0000 UTC m=+102.952649455 (1 nodes), 1.56µs from start of poll, 823.592745ms from cluster start. Timeout at 2024-07-05 13:31:02.288809098 +0000 UTC m=+103.058719773, 106.070318ms after stability -2024-07-05T13:31:02.289Z [INFO] server-1ef6ebdc-853c-1286-7841-57873c9eeb82: aborting pipeline replication: peer="{Voter server-966878d3-215d-9e09-1dee-0e44387e7554 966878d3-215d-9e09-1dee-0e44387e7554}" -2024-07-05T13:31:02.289Z [INFO] server-1ef6ebdc-853c-1286-7841-57873c9eeb82: aborting pipeline replication: peer="{Voter server-d5295a1b-2aa3-930c-ec34-7acb502246af d5295a1b-2aa3-930c-ec34-7acb502246af}" ---- PASS: TestRaft_LeaderID_Propagated (0.93s) +2024-07-05T13:37:18.130Z [INFO] server-c8d55f8f-b063-5517-6cb9-c8236df272dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8d55f8f-b063-5517-6cb9-c8236df272dd Address:c8d55f8f-b063-5517-6cb9-c8236df272dd} {Suffrage:Voter ID:server-9111b6c8-880e-14d0-d269-5470ef420412 Address:9111b6c8-880e-14d0-d269-5470ef420412} {Suffrage:Voter ID:server-cf06f9cb-c1c1-405d-59f5-eaf379ea11b9 Address:cf06f9cb-c1c1-405d-59f5-eaf379ea11b9}]" +2024-07-05T13:37:18.131Z [INFO] server-c8d55f8f-b063-5517-6cb9-c8236df272dd: entering follower state: follower="Node at c8d55f8f-b063-5517-6cb9-c8236df272dd [Follower]" leader-address= leader-id= +2024-07-05T13:37:18.131Z [INFO] server-9111b6c8-880e-14d0-d269-5470ef420412: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8d55f8f-b063-5517-6cb9-c8236df272dd Address:c8d55f8f-b063-5517-6cb9-c8236df272dd} {Suffrage:Voter ID:server-9111b6c8-880e-14d0-d269-5470ef420412 Address:9111b6c8-880e-14d0-d269-5470ef420412} {Suffrage:Voter ID:server-cf06f9cb-c1c1-405d-59f5-eaf379ea11b9 Address:cf06f9cb-c1c1-405d-59f5-eaf379ea11b9}]" +2024-07-05T13:37:18.131Z [INFO] server-9111b6c8-880e-14d0-d269-5470ef420412: entering follower state: follower="Node at 9111b6c8-880e-14d0-d269-5470ef420412 [Follower]" leader-address= leader-id= +2024-07-05T13:37:18.134Z [INFO] server-cf06f9cb-c1c1-405d-59f5-eaf379ea11b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8d55f8f-b063-5517-6cb9-c8236df272dd Address:c8d55f8f-b063-5517-6cb9-c8236df272dd} {Suffrage:Voter ID:server-9111b6c8-880e-14d0-d269-5470ef420412 Address:9111b6c8-880e-14d0-d269-5470ef420412} {Suffrage:Voter ID:server-cf06f9cb-c1c1-405d-59f5-eaf379ea11b9 Address:cf06f9cb-c1c1-405d-59f5-eaf379ea11b9}]" +2024-07-05T13:37:18.134Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:18.143Z [INFO] server-cf06f9cb-c1c1-405d-59f5-eaf379ea11b9: entering follower state: follower="Node at cf06f9cb-c1c1-405d-59f5-eaf379ea11b9 [Follower]" leader-address= leader-id= +2024-07-05T13:37:18.190Z [WARN] server-c8d55f8f-b063-5517-6cb9-c8236df272dd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:18.190Z [INFO] server-c8d55f8f-b063-5517-6cb9-c8236df272dd: entering candidate state: node="Node at c8d55f8f-b063-5517-6cb9-c8236df272dd [Candidate]" term=2 +2024-07-05T13:37:18.190Z [INFO] server-c8d55f8f-b063-5517-6cb9-c8236df272dd: election won: term=2 tally=2 +2024-07-05T13:37:18.190Z [INFO] server-c8d55f8f-b063-5517-6cb9-c8236df272dd: entering leader state: leader="Node at c8d55f8f-b063-5517-6cb9-c8236df272dd [Leader]" +2024-07-05T13:37:18.190Z [INFO] server-c8d55f8f-b063-5517-6cb9-c8236df272dd: added peer, starting replication: peer=server-9111b6c8-880e-14d0-d269-5470ef420412 +2024-07-05T13:37:18.191Z [INFO] server-c8d55f8f-b063-5517-6cb9-c8236df272dd: added peer, starting replication: peer=server-cf06f9cb-c1c1-405d-59f5-eaf379ea11b9 +2024-07-05T13:37:18.191Z [INFO] server-c8d55f8f-b063-5517-6cb9-c8236df272dd: pipelining replication: peer="{Voter server-cf06f9cb-c1c1-405d-59f5-eaf379ea11b9 cf06f9cb-c1c1-405d-59f5-eaf379ea11b9}" +2024-07-05T13:37:18.191Z [INFO] server-c8d55f8f-b063-5517-6cb9-c8236df272dd: pipelining replication: peer="{Voter server-9111b6c8-880e-14d0-d269-5470ef420412 9111b6c8-880e-14d0-d269-5470ef420412}" +2024-07-05T13:37:18.296Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:18.190999044 +0000 UTC m=+104.186919912 (1 nodes), 56.192932ms from start of poll, 60.562311ms from cluster start. Timeout at 2024-07-05 13:37:18.296047875 +0000 UTC m=+104.291968744, 105.048832ms after stability +2024-07-05T13:37:18.296Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:18.402Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:18.296522408 +0000 UTC m=+104.292443568 (1 nodes), 3.791µs from start of poll, 166.085967ms from cluster start. Timeout at 2024-07-05 13:37:18.402079314 +0000 UTC m=+104.398000766, 105.557198ms after stability +2024-07-05T13:37:18.404Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:18.510Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:18.404584685 +0000 UTC m=+104.400505845 (1 nodes), 4.083µs from start of poll, 274.148244ms from cluster start. Timeout at 2024-07-05 13:37:18.509657433 +0000 UTC m=+104.505579176, 105.073331ms after stability +2024-07-05T13:37:18.511Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:18.617Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:18.511440067 +0000 UTC m=+104.507361519 (1 nodes), 6.125µs from start of poll, 381.003918ms from cluster start. Timeout at 2024-07-05 13:37:18.616902475 +0000 UTC m=+104.612823635, 105.462116ms after stability +2024-07-05T13:37:18.617Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:18.724Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:18.618066787 +0000 UTC m=+104.613987947 (1 nodes), 4.667µs from start of poll, 487.630346ms from cluster start. Timeout at 2024-07-05 13:37:18.723987978 +0000 UTC m=+104.719909138, 105.921191ms after stability +2024-07-05T13:37:18.725Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:18.831Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:18.725567615 +0000 UTC m=+104.721488484 (1 nodes), 4.375µs from start of poll, 595.130883ms from cluster start. Timeout at 2024-07-05 13:37:18.830614405 +0000 UTC m=+104.826535274, 105.04679ms after stability +2024-07-05T13:37:18.937Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:19.043Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:18.937634868 +0000 UTC m=+104.933555445 (1 nodes), 2.917µs from start of poll, 807.197844ms from cluster start. Timeout at 2024-07-05 13:37:19.042867738 +0000 UTC m=+105.038789190, 105.233745ms after stability +2024-07-05T13:37:19.043Z [INFO] server-c8d55f8f-b063-5517-6cb9-c8236df272dd: aborting pipeline replication: peer="{Voter server-cf06f9cb-c1c1-405d-59f5-eaf379ea11b9 cf06f9cb-c1c1-405d-59f5-eaf379ea11b9}" +2024-07-05T13:37:19.043Z [INFO] server-c8d55f8f-b063-5517-6cb9-c8236df272dd: aborting pipeline replication: peer="{Voter server-9111b6c8-880e-14d0-d269-5470ef420412 9111b6c8-880e-14d0-d269-5470ef420412}" +--- PASS: TestRaft_LeaderID_Propagated (0.92s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer -2024-07-05T13:31:02.292Z [INFO] server-9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd Address:9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd} {Suffrage:Voter ID:server-ebb1b044-02de-b6d8-fa2c-58bdd47da328 Address:ebb1b044-02de-b6d8-fa2c-58bdd47da328} {Suffrage:Voter ID:server-fb79e483-3885-82fe-22fa-cf19007053e7 Address:fb79e483-3885-82fe-22fa-cf19007053e7}]" -2024-07-05T13:31:02.293Z [INFO] server-ebb1b044-02de-b6d8-fa2c-58bdd47da328: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd Address:9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd} {Suffrage:Voter ID:server-ebb1b044-02de-b6d8-fa2c-58bdd47da328 Address:ebb1b044-02de-b6d8-fa2c-58bdd47da328} {Suffrage:Voter ID:server-fb79e483-3885-82fe-22fa-cf19007053e7 Address:fb79e483-3885-82fe-22fa-cf19007053e7}]" -2024-07-05T13:31:02.293Z [INFO] server-9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd: entering follower state: follower="Node at 9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd [Follower]" leader-address= leader-id= -2024-07-05T13:31:02.293Z [INFO] server-ebb1b044-02de-b6d8-fa2c-58bdd47da328: entering follower state: follower="Node at ebb1b044-02de-b6d8-fa2c-58bdd47da328 [Follower]" leader-address= leader-id= -2024-07-05T13:31:02.293Z [INFO] server-fb79e483-3885-82fe-22fa-cf19007053e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd Address:9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd} {Suffrage:Voter ID:server-ebb1b044-02de-b6d8-fa2c-58bdd47da328 Address:ebb1b044-02de-b6d8-fa2c-58bdd47da328} {Suffrage:Voter ID:server-fb79e483-3885-82fe-22fa-cf19007053e7 Address:fb79e483-3885-82fe-22fa-cf19007053e7}]" -2024-07-05T13:31:02.293Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:02.293Z [INFO] server-fb79e483-3885-82fe-22fa-cf19007053e7: entering follower state: follower="Node at fb79e483-3885-82fe-22fa-cf19007053e7 [Follower]" leader-address= leader-id= -2024-07-05T13:31:02.369Z [WARN] server-fb79e483-3885-82fe-22fa-cf19007053e7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:02.369Z [INFO] server-fb79e483-3885-82fe-22fa-cf19007053e7: entering candidate state: node="Node at fb79e483-3885-82fe-22fa-cf19007053e7 [Candidate]" term=2 -2024-07-05T13:31:02.369Z [INFO] server-fb79e483-3885-82fe-22fa-cf19007053e7: election won: term=2 tally=2 -2024-07-05T13:31:02.369Z [INFO] server-fb79e483-3885-82fe-22fa-cf19007053e7: entering leader state: leader="Node at fb79e483-3885-82fe-22fa-cf19007053e7 [Leader]" -2024-07-05T13:31:02.370Z [INFO] server-fb79e483-3885-82fe-22fa-cf19007053e7: added peer, starting replication: peer=server-9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd -2024-07-05T13:31:02.370Z [INFO] server-fb79e483-3885-82fe-22fa-cf19007053e7: added peer, starting replication: peer=server-ebb1b044-02de-b6d8-fa2c-58bdd47da328 -2024-07-05T13:31:02.370Z [INFO] server-fb79e483-3885-82fe-22fa-cf19007053e7: pipelining replication: peer="{Voter server-ebb1b044-02de-b6d8-fa2c-58bdd47da328 ebb1b044-02de-b6d8-fa2c-58bdd47da328}" -2024-07-05T13:31:02.371Z [INFO] server-fb79e483-3885-82fe-22fa-cf19007053e7: pipelining replication: peer="{Voter server-9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd 9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd}" -2024-07-05T13:31:02.476Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:02.370033248 +0000 UTC m=+103.139943923 (1 nodes), 76.248284ms from start of poll, 78.212815ms from cluster start. Timeout at 2024-07-05 13:31:02.476217203 +0000 UTC m=+103.246127878, 106.183955ms after stability -2024-07-05T13:31:02.476Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:02.581Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:02.476411276 +0000 UTC m=+103.246321991 (1 nodes), 1.36µs from start of poll, 184.590883ms from cluster start. Timeout at 2024-07-05 13:31:02.581798659 +0000 UTC m=+103.351709334, 105.387343ms after stability -2024-07-05T13:31:02.582Z [INFO] server-9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd: entering candidate state: node="Node at 9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd [Candidate]" term=3 -2024-07-05T13:31:02.582Z [INFO] server-fb79e483-3885-82fe-22fa-cf19007053e7: entering follower state: follower="Node at fb79e483-3885-82fe-22fa-cf19007053e7 [Follower]" leader-address= leader-id= -2024-07-05T13:31:02.582Z [INFO] server-fb79e483-3885-82fe-22fa-cf19007053e7: aborting pipeline replication: peer="{Voter server-ebb1b044-02de-b6d8-fa2c-58bdd47da328 ebb1b044-02de-b6d8-fa2c-58bdd47da328}" -2024-07-05T13:31:02.582Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:02.582Z [INFO] server-9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd: election won: term=3 tally=2 -2024-07-05T13:31:02.582Z [INFO] server-9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd: entering leader state: leader="Node at 9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd [Leader]" -2024-07-05T13:31:02.582Z [INFO] server-9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd: added peer, starting replication: peer=server-ebb1b044-02de-b6d8-fa2c-58bdd47da328 -2024-07-05T13:31:02.582Z [INFO] server-9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd: added peer, starting replication: peer=server-fb79e483-3885-82fe-22fa-cf19007053e7 -2024-07-05T13:31:02.582Z [INFO] server-fb79e483-3885-82fe-22fa-cf19007053e7: aborting pipeline replication: peer="{Voter server-9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd 9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd}" -2024-07-05T13:31:02.582Z [INFO] server-9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd: pipelining replication: peer="{Voter server-ebb1b044-02de-b6d8-fa2c-58bdd47da328 ebb1b044-02de-b6d8-fa2c-58bdd47da328}" -2024-07-05T13:31:02.582Z [INFO] server-9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd: pipelining replication: peer="{Voter server-fb79e483-3885-82fe-22fa-cf19007053e7 fb79e483-3885-82fe-22fa-cf19007053e7}" -2024-07-05T13:31:02.688Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:02.582748345 +0000 UTC m=+103.352659020 (1 nodes), 387.586µs from start of poll, 290.927912ms from cluster start. Timeout at 2024-07-05 13:31:02.688424998 +0000 UTC m=+103.458335673, 105.676653ms after stability -2024-07-05T13:31:02.688Z [INFO] server-9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd: aborting pipeline replication: peer="{Voter server-fb79e483-3885-82fe-22fa-cf19007053e7 fb79e483-3885-82fe-22fa-cf19007053e7}" -2024-07-05T13:31:02.688Z [INFO] server-9a1e3de8-c7e0-dd8b-79c1-1da0c9268ebd: aborting pipeline replication: peer="{Voter server-ebb1b044-02de-b6d8-fa2c-58bdd47da328 ebb1b044-02de-b6d8-fa2c-58bdd47da328}" ---- PASS: TestRaft_LeadershipTransfer (0.40s) +2024-07-05T13:37:19.064Z [INFO] server-f42940f1-f205-e32c-c62b-2c801cbf7421: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f42940f1-f205-e32c-c62b-2c801cbf7421 Address:f42940f1-f205-e32c-c62b-2c801cbf7421} {Suffrage:Voter ID:server-daa45067-d506-a66f-5ecd-a46e9387d62a Address:daa45067-d506-a66f-5ecd-a46e9387d62a} {Suffrage:Voter ID:server-b6134699-003b-711a-bcf9-7ab13fa25682 Address:b6134699-003b-711a-bcf9-7ab13fa25682}]" +2024-07-05T13:37:19.064Z [INFO] server-f42940f1-f205-e32c-c62b-2c801cbf7421: entering follower state: follower="Node at f42940f1-f205-e32c-c62b-2c801cbf7421 [Follower]" leader-address= leader-id= +2024-07-05T13:37:19.064Z [INFO] server-daa45067-d506-a66f-5ecd-a46e9387d62a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f42940f1-f205-e32c-c62b-2c801cbf7421 Address:f42940f1-f205-e32c-c62b-2c801cbf7421} {Suffrage:Voter ID:server-daa45067-d506-a66f-5ecd-a46e9387d62a Address:daa45067-d506-a66f-5ecd-a46e9387d62a} {Suffrage:Voter ID:server-b6134699-003b-711a-bcf9-7ab13fa25682 Address:b6134699-003b-711a-bcf9-7ab13fa25682}]" +2024-07-05T13:37:19.065Z [INFO] server-daa45067-d506-a66f-5ecd-a46e9387d62a: entering follower state: follower="Node at daa45067-d506-a66f-5ecd-a46e9387d62a [Follower]" leader-address= leader-id= +2024-07-05T13:37:19.065Z [INFO] server-b6134699-003b-711a-bcf9-7ab13fa25682: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f42940f1-f205-e32c-c62b-2c801cbf7421 Address:f42940f1-f205-e32c-c62b-2c801cbf7421} {Suffrage:Voter ID:server-daa45067-d506-a66f-5ecd-a46e9387d62a Address:daa45067-d506-a66f-5ecd-a46e9387d62a} {Suffrage:Voter ID:server-b6134699-003b-711a-bcf9-7ab13fa25682 Address:b6134699-003b-711a-bcf9-7ab13fa25682}]" +2024-07-05T13:37:19.065Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:19.065Z [INFO] server-b6134699-003b-711a-bcf9-7ab13fa25682: entering follower state: follower="Node at b6134699-003b-711a-bcf9-7ab13fa25682 [Follower]" leader-address= leader-id= +2024-07-05T13:37:19.122Z [WARN] server-f42940f1-f205-e32c-c62b-2c801cbf7421: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:19.123Z [INFO] server-f42940f1-f205-e32c-c62b-2c801cbf7421: entering candidate state: node="Node at f42940f1-f205-e32c-c62b-2c801cbf7421 [Candidate]" term=2 +2024-07-05T13:37:19.123Z [INFO] server-f42940f1-f205-e32c-c62b-2c801cbf7421: election won: term=2 tally=2 +2024-07-05T13:37:19.123Z [INFO] server-f42940f1-f205-e32c-c62b-2c801cbf7421: entering leader state: leader="Node at f42940f1-f205-e32c-c62b-2c801cbf7421 [Leader]" +2024-07-05T13:37:19.123Z [INFO] server-f42940f1-f205-e32c-c62b-2c801cbf7421: added peer, starting replication: peer=server-daa45067-d506-a66f-5ecd-a46e9387d62a +2024-07-05T13:37:19.123Z [INFO] server-f42940f1-f205-e32c-c62b-2c801cbf7421: added peer, starting replication: peer=server-b6134699-003b-711a-bcf9-7ab13fa25682 +2024-07-05T13:37:19.124Z [INFO] server-f42940f1-f205-e32c-c62b-2c801cbf7421: pipelining replication: peer="{Voter server-daa45067-d506-a66f-5ecd-a46e9387d62a daa45067-d506-a66f-5ecd-a46e9387d62a}" +2024-07-05T13:37:19.124Z [INFO] server-f42940f1-f205-e32c-c62b-2c801cbf7421: pipelining replication: peer="{Voter server-b6134699-003b-711a-bcf9-7ab13fa25682 b6134699-003b-711a-bcf9-7ab13fa25682}" +2024-07-05T13:37:19.228Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:19.123813676 +0000 UTC m=+105.119734836 (1 nodes), 57.920442ms from start of poll, 60.077861ms from cluster start. Timeout at 2024-07-05 13:37:19.228849092 +0000 UTC m=+105.224769960, 105.035124ms after stability +2024-07-05T13:37:19.229Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:19.334Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:19.229162044 +0000 UTC m=+105.225082913 (1 nodes), 3.792µs from start of poll, 165.425938ms from cluster start. Timeout at 2024-07-05 13:37:19.334408914 +0000 UTC m=+105.330330074, 105.247161ms after stability +2024-07-05T13:37:19.334Z [INFO] server-daa45067-d506-a66f-5ecd-a46e9387d62a: entering candidate state: node="Node at daa45067-d506-a66f-5ecd-a46e9387d62a [Candidate]" term=3 +2024-07-05T13:37:19.335Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:19.335Z [INFO] server-f42940f1-f205-e32c-c62b-2c801cbf7421: entering follower state: follower="Node at f42940f1-f205-e32c-c62b-2c801cbf7421 [Follower]" leader-address= leader-id= +2024-07-05T13:37:19.335Z [INFO] server-daa45067-d506-a66f-5ecd-a46e9387d62a: election won: term=3 tally=2 +2024-07-05T13:37:19.335Z [INFO] server-f42940f1-f205-e32c-c62b-2c801cbf7421: aborting pipeline replication: peer="{Voter server-daa45067-d506-a66f-5ecd-a46e9387d62a daa45067-d506-a66f-5ecd-a46e9387d62a}" +2024-07-05T13:37:19.335Z [INFO] server-daa45067-d506-a66f-5ecd-a46e9387d62a: entering leader state: leader="Node at daa45067-d506-a66f-5ecd-a46e9387d62a [Leader]" +2024-07-05T13:37:19.335Z [INFO] server-f42940f1-f205-e32c-c62b-2c801cbf7421: aborting pipeline replication: peer="{Voter server-b6134699-003b-711a-bcf9-7ab13fa25682 b6134699-003b-711a-bcf9-7ab13fa25682}" +2024-07-05T13:37:19.335Z [INFO] server-daa45067-d506-a66f-5ecd-a46e9387d62a: added peer, starting replication: peer=server-f42940f1-f205-e32c-c62b-2c801cbf7421 +2024-07-05T13:37:19.335Z [INFO] server-daa45067-d506-a66f-5ecd-a46e9387d62a: added peer, starting replication: peer=server-b6134699-003b-711a-bcf9-7ab13fa25682 +2024-07-05T13:37:19.336Z [INFO] server-daa45067-d506-a66f-5ecd-a46e9387d62a: pipelining replication: peer="{Voter server-f42940f1-f205-e32c-c62b-2c801cbf7421 f42940f1-f205-e32c-c62b-2c801cbf7421}" +2024-07-05T13:37:19.336Z [INFO] server-daa45067-d506-a66f-5ecd-a46e9387d62a: pipelining replication: peer="{Voter server-b6134699-003b-711a-bcf9-7ab13fa25682 b6134699-003b-711a-bcf9-7ab13fa25682}" +2024-07-05T13:37:19.440Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:19.33563535 +0000 UTC m=+105.331556218 (1 nodes), 539.281µs from start of poll, 271.899243ms from cluster start. Timeout at 2024-07-05 13:37:19.440680099 +0000 UTC m=+105.436601550, 105.045332ms after stability +2024-07-05T13:37:19.441Z [INFO] server-daa45067-d506-a66f-5ecd-a46e9387d62a: aborting pipeline replication: peer="{Voter server-b6134699-003b-711a-bcf9-7ab13fa25682 b6134699-003b-711a-bcf9-7ab13fa25682}" +2024-07-05T13:37:19.441Z [INFO] server-daa45067-d506-a66f-5ecd-a46e9387d62a: aborting pipeline replication: peer="{Voter server-f42940f1-f205-e32c-c62b-2c801cbf7421 f42940f1-f205-e32c-c62b-2c801cbf7421}" +--- PASS: TestRaft_LeadershipTransfer (0.39s) === RUN TestRaft_LeadershipTransferWithOneNode -2024-07-05T13:31:02.690Z [INFO] server-bdaea630-b8ac-1138-0513-6d9db94b8840: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bdaea630-b8ac-1138-0513-6d9db94b8840 Address:bdaea630-b8ac-1138-0513-6d9db94b8840}]" -2024-07-05T13:31:02.690Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:02.690Z [INFO] server-bdaea630-b8ac-1138-0513-6d9db94b8840: entering follower state: follower="Node at bdaea630-b8ac-1138-0513-6d9db94b8840 [Follower]" leader-address= leader-id= -2024-07-05T13:31:02.752Z [WARN] server-bdaea630-b8ac-1138-0513-6d9db94b8840: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:02.752Z [INFO] server-bdaea630-b8ac-1138-0513-6d9db94b8840: entering candidate state: node="Node at bdaea630-b8ac-1138-0513-6d9db94b8840 [Candidate]" term=2 -2024-07-05T13:31:02.752Z [INFO] server-bdaea630-b8ac-1138-0513-6d9db94b8840: election won: term=2 tally=1 -2024-07-05T13:31:02.752Z [INFO] server-bdaea630-b8ac-1138-0513-6d9db94b8840: entering leader state: leader="Node at bdaea630-b8ac-1138-0513-6d9db94b8840 [Leader]" -2024-07-05T13:31:02.858Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:02.752980613 +0000 UTC m=+103.522891288 (1 nodes), 62.216057ms from start of poll, 62.515287ms from cluster start. Timeout at 2024-07-05 13:31:02.858088085 +0000 UTC m=+103.627998760, 105.107472ms after stability ---- PASS: TestRaft_LeadershipTransferWithOneNode (0.17s) +2024-07-05T13:37:19.451Z [INFO] server-2450ab23-36e8-6ae6-b85e-d520c0f5f485: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2450ab23-36e8-6ae6-b85e-d520c0f5f485 Address:2450ab23-36e8-6ae6-b85e-d520c0f5f485}]" +2024-07-05T13:37:19.452Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:19.452Z [INFO] server-2450ab23-36e8-6ae6-b85e-d520c0f5f485: entering follower state: follower="Node at 2450ab23-36e8-6ae6-b85e-d520c0f5f485 [Follower]" leader-address= leader-id= +2024-07-05T13:37:19.529Z [WARN] server-2450ab23-36e8-6ae6-b85e-d520c0f5f485: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:19.529Z [INFO] server-2450ab23-36e8-6ae6-b85e-d520c0f5f485: entering candidate state: node="Node at 2450ab23-36e8-6ae6-b85e-d520c0f5f485 [Candidate]" term=2 +2024-07-05T13:37:19.530Z [INFO] server-2450ab23-36e8-6ae6-b85e-d520c0f5f485: election won: term=2 tally=1 +2024-07-05T13:37:19.530Z [INFO] server-2450ab23-36e8-6ae6-b85e-d520c0f5f485: entering leader state: leader="Node at 2450ab23-36e8-6ae6-b85e-d520c0f5f485 [Leader]" +2024-07-05T13:37:19.641Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:19.536191604 +0000 UTC m=+105.532112764 (1 nodes), 83.995257ms from start of poll, 84.685036ms from cluster start. Timeout at 2024-07-05 13:37:19.641290019 +0000 UTC m=+105.637211762, 105.098998ms after stability +--- PASS: TestRaft_LeadershipTransferWithOneNode (0.19s) === RUN TestRaft_LeadershipTransferWithSevenNodes -2024-07-05T13:31:02.863Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c4558ca-7e57-9752-7cfa-1af0608dc63e Address:3c4558ca-7e57-9752-7cfa-1af0608dc63e} {Suffrage:Voter ID:server-4b50b308-fb55-390d-ed40-7ee952e21d17 Address:4b50b308-fb55-390d-ed40-7ee952e21d17} {Suffrage:Voter ID:server-1567b92b-774f-1de6-3ef3-aec76d361788 Address:1567b92b-774f-1de6-3ef3-aec76d361788} {Suffrage:Voter ID:server-4fb4936b-7550-35d5-7d51-0e9ebd67d162 Address:4fb4936b-7550-35d5-7d51-0e9ebd67d162} {Suffrage:Voter ID:server-23ea3661-d9c1-d276-6ab7-7212086b5965 Address:23ea3661-d9c1-d276-6ab7-7212086b5965} {Suffrage:Voter ID:server-914a11ed-0951-6003-1f14-ff3ef0717f14 Address:914a11ed-0951-6003-1f14-ff3ef0717f14} {Suffrage:Voter ID:server-8c20fbb1-196e-43fb-52db-273f4fc965d1 Address:8c20fbb1-196e-43fb-52db-273f4fc965d1}]" -2024-07-05T13:31:02.863Z [INFO] server-4b50b308-fb55-390d-ed40-7ee952e21d17: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c4558ca-7e57-9752-7cfa-1af0608dc63e Address:3c4558ca-7e57-9752-7cfa-1af0608dc63e} {Suffrage:Voter ID:server-4b50b308-fb55-390d-ed40-7ee952e21d17 Address:4b50b308-fb55-390d-ed40-7ee952e21d17} {Suffrage:Voter ID:server-1567b92b-774f-1de6-3ef3-aec76d361788 Address:1567b92b-774f-1de6-3ef3-aec76d361788} {Suffrage:Voter ID:server-4fb4936b-7550-35d5-7d51-0e9ebd67d162 Address:4fb4936b-7550-35d5-7d51-0e9ebd67d162} {Suffrage:Voter ID:server-23ea3661-d9c1-d276-6ab7-7212086b5965 Address:23ea3661-d9c1-d276-6ab7-7212086b5965} {Suffrage:Voter ID:server-914a11ed-0951-6003-1f14-ff3ef0717f14 Address:914a11ed-0951-6003-1f14-ff3ef0717f14} {Suffrage:Voter ID:server-8c20fbb1-196e-43fb-52db-273f4fc965d1 Address:8c20fbb1-196e-43fb-52db-273f4fc965d1}]" -2024-07-05T13:31:02.863Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c4558ca-7e57-9752-7cfa-1af0608dc63e Address:3c4558ca-7e57-9752-7cfa-1af0608dc63e} {Suffrage:Voter ID:server-4b50b308-fb55-390d-ed40-7ee952e21d17 Address:4b50b308-fb55-390d-ed40-7ee952e21d17} {Suffrage:Voter ID:server-1567b92b-774f-1de6-3ef3-aec76d361788 Address:1567b92b-774f-1de6-3ef3-aec76d361788} {Suffrage:Voter ID:server-4fb4936b-7550-35d5-7d51-0e9ebd67d162 Address:4fb4936b-7550-35d5-7d51-0e9ebd67d162} {Suffrage:Voter ID:server-23ea3661-d9c1-d276-6ab7-7212086b5965 Address:23ea3661-d9c1-d276-6ab7-7212086b5965} {Suffrage:Voter ID:server-914a11ed-0951-6003-1f14-ff3ef0717f14 Address:914a11ed-0951-6003-1f14-ff3ef0717f14} {Suffrage:Voter ID:server-8c20fbb1-196e-43fb-52db-273f4fc965d1 Address:8c20fbb1-196e-43fb-52db-273f4fc965d1}]" -2024-07-05T13:31:02.864Z [INFO] server-4b50b308-fb55-390d-ed40-7ee952e21d17: entering follower state: follower="Node at 4b50b308-fb55-390d-ed40-7ee952e21d17 [Follower]" leader-address= leader-id= -2024-07-05T13:31:02.864Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: entering follower state: follower="Node at 3c4558ca-7e57-9752-7cfa-1af0608dc63e [Follower]" leader-address= leader-id= -2024-07-05T13:31:02.864Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: entering follower state: follower="Node at 1567b92b-774f-1de6-3ef3-aec76d361788 [Follower]" leader-address= leader-id= -2024-07-05T13:31:02.868Z [INFO] server-4fb4936b-7550-35d5-7d51-0e9ebd67d162: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c4558ca-7e57-9752-7cfa-1af0608dc63e Address:3c4558ca-7e57-9752-7cfa-1af0608dc63e} {Suffrage:Voter ID:server-4b50b308-fb55-390d-ed40-7ee952e21d17 Address:4b50b308-fb55-390d-ed40-7ee952e21d17} {Suffrage:Voter ID:server-1567b92b-774f-1de6-3ef3-aec76d361788 Address:1567b92b-774f-1de6-3ef3-aec76d361788} {Suffrage:Voter ID:server-4fb4936b-7550-35d5-7d51-0e9ebd67d162 Address:4fb4936b-7550-35d5-7d51-0e9ebd67d162} {Suffrage:Voter ID:server-23ea3661-d9c1-d276-6ab7-7212086b5965 Address:23ea3661-d9c1-d276-6ab7-7212086b5965} {Suffrage:Voter ID:server-914a11ed-0951-6003-1f14-ff3ef0717f14 Address:914a11ed-0951-6003-1f14-ff3ef0717f14} {Suffrage:Voter ID:server-8c20fbb1-196e-43fb-52db-273f4fc965d1 Address:8c20fbb1-196e-43fb-52db-273f4fc965d1}]" -2024-07-05T13:31:02.869Z [INFO] server-23ea3661-d9c1-d276-6ab7-7212086b5965: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c4558ca-7e57-9752-7cfa-1af0608dc63e Address:3c4558ca-7e57-9752-7cfa-1af0608dc63e} {Suffrage:Voter ID:server-4b50b308-fb55-390d-ed40-7ee952e21d17 Address:4b50b308-fb55-390d-ed40-7ee952e21d17} {Suffrage:Voter ID:server-1567b92b-774f-1de6-3ef3-aec76d361788 Address:1567b92b-774f-1de6-3ef3-aec76d361788} {Suffrage:Voter ID:server-4fb4936b-7550-35d5-7d51-0e9ebd67d162 Address:4fb4936b-7550-35d5-7d51-0e9ebd67d162} {Suffrage:Voter ID:server-23ea3661-d9c1-d276-6ab7-7212086b5965 Address:23ea3661-d9c1-d276-6ab7-7212086b5965} {Suffrage:Voter ID:server-914a11ed-0951-6003-1f14-ff3ef0717f14 Address:914a11ed-0951-6003-1f14-ff3ef0717f14} {Suffrage:Voter ID:server-8c20fbb1-196e-43fb-52db-273f4fc965d1 Address:8c20fbb1-196e-43fb-52db-273f4fc965d1}]" -2024-07-05T13:31:02.872Z [INFO] server-23ea3661-d9c1-d276-6ab7-7212086b5965: entering follower state: follower="Node at 23ea3661-d9c1-d276-6ab7-7212086b5965 [Follower]" leader-address= leader-id= -2024-07-05T13:31:02.872Z [INFO] server-4fb4936b-7550-35d5-7d51-0e9ebd67d162: entering follower state: follower="Node at 4fb4936b-7550-35d5-7d51-0e9ebd67d162 [Follower]" leader-address= leader-id= -2024-07-05T13:31:02.879Z [INFO] server-914a11ed-0951-6003-1f14-ff3ef0717f14: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c4558ca-7e57-9752-7cfa-1af0608dc63e Address:3c4558ca-7e57-9752-7cfa-1af0608dc63e} {Suffrage:Voter ID:server-4b50b308-fb55-390d-ed40-7ee952e21d17 Address:4b50b308-fb55-390d-ed40-7ee952e21d17} {Suffrage:Voter ID:server-1567b92b-774f-1de6-3ef3-aec76d361788 Address:1567b92b-774f-1de6-3ef3-aec76d361788} {Suffrage:Voter ID:server-4fb4936b-7550-35d5-7d51-0e9ebd67d162 Address:4fb4936b-7550-35d5-7d51-0e9ebd67d162} {Suffrage:Voter ID:server-23ea3661-d9c1-d276-6ab7-7212086b5965 Address:23ea3661-d9c1-d276-6ab7-7212086b5965} {Suffrage:Voter ID:server-914a11ed-0951-6003-1f14-ff3ef0717f14 Address:914a11ed-0951-6003-1f14-ff3ef0717f14} {Suffrage:Voter ID:server-8c20fbb1-196e-43fb-52db-273f4fc965d1 Address:8c20fbb1-196e-43fb-52db-273f4fc965d1}]" -2024-07-05T13:31:02.880Z [INFO] server-8c20fbb1-196e-43fb-52db-273f4fc965d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c4558ca-7e57-9752-7cfa-1af0608dc63e Address:3c4558ca-7e57-9752-7cfa-1af0608dc63e} {Suffrage:Voter ID:server-4b50b308-fb55-390d-ed40-7ee952e21d17 Address:4b50b308-fb55-390d-ed40-7ee952e21d17} {Suffrage:Voter ID:server-1567b92b-774f-1de6-3ef3-aec76d361788 Address:1567b92b-774f-1de6-3ef3-aec76d361788} {Suffrage:Voter ID:server-4fb4936b-7550-35d5-7d51-0e9ebd67d162 Address:4fb4936b-7550-35d5-7d51-0e9ebd67d162} {Suffrage:Voter ID:server-23ea3661-d9c1-d276-6ab7-7212086b5965 Address:23ea3661-d9c1-d276-6ab7-7212086b5965} {Suffrage:Voter ID:server-914a11ed-0951-6003-1f14-ff3ef0717f14 Address:914a11ed-0951-6003-1f14-ff3ef0717f14} {Suffrage:Voter ID:server-8c20fbb1-196e-43fb-52db-273f4fc965d1 Address:8c20fbb1-196e-43fb-52db-273f4fc965d1}]" -2024-07-05T13:31:02.880Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:02.880Z [INFO] server-914a11ed-0951-6003-1f14-ff3ef0717f14: entering follower state: follower="Node at 914a11ed-0951-6003-1f14-ff3ef0717f14 [Follower]" leader-address= leader-id= -2024-07-05T13:31:02.880Z [INFO] server-8c20fbb1-196e-43fb-52db-273f4fc965d1: entering follower state: follower="Node at 8c20fbb1-196e-43fb-52db-273f4fc965d1 [Follower]" leader-address= leader-id= -2024-07-05T13:31:02.940Z [WARN] server-1567b92b-774f-1de6-3ef3-aec76d361788: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:02.940Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: entering candidate state: node="Node at 1567b92b-774f-1de6-3ef3-aec76d361788 [Candidate]" term=2 -2024-07-05T13:31:02.940Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: election won: term=2 tally=4 -2024-07-05T13:31:02.940Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: entering leader state: leader="Node at 1567b92b-774f-1de6-3ef3-aec76d361788 [Leader]" -2024-07-05T13:31:02.940Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: added peer, starting replication: peer=server-3c4558ca-7e57-9752-7cfa-1af0608dc63e -2024-07-05T13:31:02.940Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: added peer, starting replication: peer=server-4b50b308-fb55-390d-ed40-7ee952e21d17 -2024-07-05T13:31:02.940Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: added peer, starting replication: peer=server-4fb4936b-7550-35d5-7d51-0e9ebd67d162 -2024-07-05T13:31:02.940Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: added peer, starting replication: peer=server-23ea3661-d9c1-d276-6ab7-7212086b5965 -2024-07-05T13:31:02.940Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: added peer, starting replication: peer=server-914a11ed-0951-6003-1f14-ff3ef0717f14 -2024-07-05T13:31:02.940Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: added peer, starting replication: peer=server-8c20fbb1-196e-43fb-52db-273f4fc965d1 -2024-07-05T13:31:02.941Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: pipelining replication: peer="{Voter server-8c20fbb1-196e-43fb-52db-273f4fc965d1 8c20fbb1-196e-43fb-52db-273f4fc965d1}" -2024-07-05T13:31:02.941Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: pipelining replication: peer="{Voter server-3c4558ca-7e57-9752-7cfa-1af0608dc63e 3c4558ca-7e57-9752-7cfa-1af0608dc63e}" -2024-07-05T13:31:02.941Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: pipelining replication: peer="{Voter server-914a11ed-0951-6003-1f14-ff3ef0717f14 914a11ed-0951-6003-1f14-ff3ef0717f14}" -2024-07-05T13:31:02.941Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: pipelining replication: peer="{Voter server-4fb4936b-7550-35d5-7d51-0e9ebd67d162 4fb4936b-7550-35d5-7d51-0e9ebd67d162}" -2024-07-05T13:31:02.941Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: pipelining replication: peer="{Voter server-23ea3661-d9c1-d276-6ab7-7212086b5965 23ea3661-d9c1-d276-6ab7-7212086b5965}" -2024-07-05T13:31:02.941Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: pipelining replication: peer="{Voter server-4b50b308-fb55-390d-ed40-7ee952e21d17 4b50b308-fb55-390d-ed40-7ee952e21d17}" -2024-07-05T13:31:03.049Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:02.941687632 +0000 UTC m=+103.711598307 (1 nodes), 61.257771ms from start of poll, 78.589403ms from cluster start. Timeout at 2024-07-05 13:31:03.048764275 +0000 UTC m=+103.818674990, 107.076683ms after stability -2024-07-05T13:31:03.049Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:31:03.154Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:31:03.049531049 +0000 UTC m=+103.819441724 (6 nodes), 5.84µs from start of poll, 186.43282ms from cluster start. Timeout at 2024-07-05 13:31:03.154541484 +0000 UTC m=+103.924452159, 105.010435ms after stability -2024-07-05T13:31:03.154Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:03.262Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:03.154695039 +0000 UTC m=+103.924605714 (1 nodes), 2.6µs from start of poll, 291.59681ms from cluster start. Timeout at 2024-07-05 13:31:03.262158829 +0000 UTC m=+104.032069504, 107.46379ms after stability -2024-07-05T13:31:03.262Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: entering candidate state: node="Node at 3c4558ca-7e57-9752-7cfa-1af0608dc63e [Candidate]" term=3 -2024-07-05T13:31:03.262Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:03.262Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: election won: term=3 tally=4 -2024-07-05T13:31:03.262Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: entering leader state: leader="Node at 3c4558ca-7e57-9752-7cfa-1af0608dc63e [Leader]" -2024-07-05T13:31:03.262Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: added peer, starting replication: peer=server-4b50b308-fb55-390d-ed40-7ee952e21d17 -2024-07-05T13:31:03.262Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: added peer, starting replication: peer=server-1567b92b-774f-1de6-3ef3-aec76d361788 -2024-07-05T13:31:03.262Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: entering follower state: follower="Node at 1567b92b-774f-1de6-3ef3-aec76d361788 [Follower]" leader-address= leader-id= -2024-07-05T13:31:03.262Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: added peer, starting replication: peer=server-4fb4936b-7550-35d5-7d51-0e9ebd67d162 -2024-07-05T13:31:03.263Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: added peer, starting replication: peer=server-23ea3661-d9c1-d276-6ab7-7212086b5965 -2024-07-05T13:31:03.263Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: added peer, starting replication: peer=server-914a11ed-0951-6003-1f14-ff3ef0717f14 -2024-07-05T13:31:03.263Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: added peer, starting replication: peer=server-8c20fbb1-196e-43fb-52db-273f4fc965d1 -2024-07-05T13:31:03.263Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: pipelining replication: peer="{Voter server-1567b92b-774f-1de6-3ef3-aec76d361788 1567b92b-774f-1de6-3ef3-aec76d361788}" -2024-07-05T13:31:03.263Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: pipelining replication: peer="{Voter server-8c20fbb1-196e-43fb-52db-273f4fc965d1 8c20fbb1-196e-43fb-52db-273f4fc965d1}" -2024-07-05T13:31:03.263Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: aborting pipeline replication: peer="{Voter server-4fb4936b-7550-35d5-7d51-0e9ebd67d162 4fb4936b-7550-35d5-7d51-0e9ebd67d162}" -2024-07-05T13:31:03.263Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: aborting pipeline replication: peer="{Voter server-23ea3661-d9c1-d276-6ab7-7212086b5965 23ea3661-d9c1-d276-6ab7-7212086b5965}" -2024-07-05T13:31:03.263Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: pipelining replication: peer="{Voter server-4b50b308-fb55-390d-ed40-7ee952e21d17 4b50b308-fb55-390d-ed40-7ee952e21d17}" -2024-07-05T13:31:03.263Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: aborting pipeline replication: peer="{Voter server-914a11ed-0951-6003-1f14-ff3ef0717f14 914a11ed-0951-6003-1f14-ff3ef0717f14}" -2024-07-05T13:31:03.263Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: aborting pipeline replication: peer="{Voter server-8c20fbb1-196e-43fb-52db-273f4fc965d1 8c20fbb1-196e-43fb-52db-273f4fc965d1}" -2024-07-05T13:31:03.263Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: aborting pipeline replication: peer="{Voter server-3c4558ca-7e57-9752-7cfa-1af0608dc63e 3c4558ca-7e57-9752-7cfa-1af0608dc63e}" -2024-07-05T13:31:03.263Z [INFO] server-1567b92b-774f-1de6-3ef3-aec76d361788: aborting pipeline replication: peer="{Voter server-4b50b308-fb55-390d-ed40-7ee952e21d17 4b50b308-fb55-390d-ed40-7ee952e21d17}" -2024-07-05T13:31:03.263Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: pipelining replication: peer="{Voter server-4fb4936b-7550-35d5-7d51-0e9ebd67d162 4fb4936b-7550-35d5-7d51-0e9ebd67d162}" -2024-07-05T13:31:03.263Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: pipelining replication: peer="{Voter server-23ea3661-d9c1-d276-6ab7-7212086b5965 23ea3661-d9c1-d276-6ab7-7212086b5965}" -2024-07-05T13:31:03.264Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: pipelining replication: peer="{Voter server-914a11ed-0951-6003-1f14-ff3ef0717f14 914a11ed-0951-6003-1f14-ff3ef0717f14}" -2024-07-05T13:31:03.368Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:03.263313228 +0000 UTC m=+104.033223903 (1 nodes), 569.82µs from start of poll, 400.214999ms from cluster start. Timeout at 2024-07-05 13:31:03.368637613 +0000 UTC m=+104.138548288, 105.324385ms after stability -2024-07-05T13:31:03.368Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: aborting pipeline replication: peer="{Voter server-8c20fbb1-196e-43fb-52db-273f4fc965d1 8c20fbb1-196e-43fb-52db-273f4fc965d1}" -2024-07-05T13:31:03.368Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: aborting pipeline replication: peer="{Voter server-1567b92b-774f-1de6-3ef3-aec76d361788 1567b92b-774f-1de6-3ef3-aec76d361788}" -2024-07-05T13:31:03.368Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: aborting pipeline replication: peer="{Voter server-23ea3661-d9c1-d276-6ab7-7212086b5965 23ea3661-d9c1-d276-6ab7-7212086b5965}" -2024-07-05T13:31:03.368Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: aborting pipeline replication: peer="{Voter server-4b50b308-fb55-390d-ed40-7ee952e21d17 4b50b308-fb55-390d-ed40-7ee952e21d17}" -2024-07-05T13:31:03.368Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: aborting pipeline replication: peer="{Voter server-914a11ed-0951-6003-1f14-ff3ef0717f14 914a11ed-0951-6003-1f14-ff3ef0717f14}" -2024-07-05T13:31:03.368Z [INFO] server-3c4558ca-7e57-9752-7cfa-1af0608dc63e: aborting pipeline replication: peer="{Voter server-4fb4936b-7550-35d5-7d51-0e9ebd67d162 4fb4936b-7550-35d5-7d51-0e9ebd67d162}" ---- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.51s) +2024-07-05T13:37:19.655Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5e00496-673e-2be1-18e6-162ad07b74af Address:d5e00496-673e-2be1-18e6-162ad07b74af} {Suffrage:Voter ID:server-22d988fe-44ff-7988-d25e-53384ef3638f Address:22d988fe-44ff-7988-d25e-53384ef3638f} {Suffrage:Voter ID:server-1af0a5fb-ff75-25a5-a673-386a8d819f02 Address:1af0a5fb-ff75-25a5-a673-386a8d819f02} {Suffrage:Voter ID:server-5b1deb0c-822b-e546-0b2c-945d89676f38 Address:5b1deb0c-822b-e546-0b2c-945d89676f38} {Suffrage:Voter ID:server-566b03f7-4507-e631-dde7-f1d7a041aa77 Address:566b03f7-4507-e631-dde7-f1d7a041aa77} {Suffrage:Voter ID:server-3dc64780-c8e3-d590-e146-de5c1a67eba7 Address:3dc64780-c8e3-d590-e146-de5c1a67eba7} {Suffrage:Voter ID:server-0d905eac-482d-e083-508f-a91a1fbc51f2 Address:0d905eac-482d-e083-508f-a91a1fbc51f2}]" +2024-07-05T13:37:19.655Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: entering follower state: follower="Node at d5e00496-673e-2be1-18e6-162ad07b74af [Follower]" leader-address= leader-id= +2024-07-05T13:37:19.656Z [INFO] server-22d988fe-44ff-7988-d25e-53384ef3638f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5e00496-673e-2be1-18e6-162ad07b74af Address:d5e00496-673e-2be1-18e6-162ad07b74af} {Suffrage:Voter ID:server-22d988fe-44ff-7988-d25e-53384ef3638f Address:22d988fe-44ff-7988-d25e-53384ef3638f} {Suffrage:Voter ID:server-1af0a5fb-ff75-25a5-a673-386a8d819f02 Address:1af0a5fb-ff75-25a5-a673-386a8d819f02} {Suffrage:Voter ID:server-5b1deb0c-822b-e546-0b2c-945d89676f38 Address:5b1deb0c-822b-e546-0b2c-945d89676f38} {Suffrage:Voter ID:server-566b03f7-4507-e631-dde7-f1d7a041aa77 Address:566b03f7-4507-e631-dde7-f1d7a041aa77} {Suffrage:Voter ID:server-3dc64780-c8e3-d590-e146-de5c1a67eba7 Address:3dc64780-c8e3-d590-e146-de5c1a67eba7} {Suffrage:Voter ID:server-0d905eac-482d-e083-508f-a91a1fbc51f2 Address:0d905eac-482d-e083-508f-a91a1fbc51f2}]" +2024-07-05T13:37:19.656Z [INFO] server-22d988fe-44ff-7988-d25e-53384ef3638f: entering follower state: follower="Node at 22d988fe-44ff-7988-d25e-53384ef3638f [Follower]" leader-address= leader-id= +2024-07-05T13:37:19.657Z [INFO] server-1af0a5fb-ff75-25a5-a673-386a8d819f02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5e00496-673e-2be1-18e6-162ad07b74af Address:d5e00496-673e-2be1-18e6-162ad07b74af} {Suffrage:Voter ID:server-22d988fe-44ff-7988-d25e-53384ef3638f Address:22d988fe-44ff-7988-d25e-53384ef3638f} {Suffrage:Voter ID:server-1af0a5fb-ff75-25a5-a673-386a8d819f02 Address:1af0a5fb-ff75-25a5-a673-386a8d819f02} {Suffrage:Voter ID:server-5b1deb0c-822b-e546-0b2c-945d89676f38 Address:5b1deb0c-822b-e546-0b2c-945d89676f38} {Suffrage:Voter ID:server-566b03f7-4507-e631-dde7-f1d7a041aa77 Address:566b03f7-4507-e631-dde7-f1d7a041aa77} {Suffrage:Voter ID:server-3dc64780-c8e3-d590-e146-de5c1a67eba7 Address:3dc64780-c8e3-d590-e146-de5c1a67eba7} {Suffrage:Voter ID:server-0d905eac-482d-e083-508f-a91a1fbc51f2 Address:0d905eac-482d-e083-508f-a91a1fbc51f2}]" +2024-07-05T13:37:19.657Z [INFO] server-1af0a5fb-ff75-25a5-a673-386a8d819f02: entering follower state: follower="Node at 1af0a5fb-ff75-25a5-a673-386a8d819f02 [Follower]" leader-address= leader-id= +2024-07-05T13:37:19.657Z [INFO] server-5b1deb0c-822b-e546-0b2c-945d89676f38: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5e00496-673e-2be1-18e6-162ad07b74af Address:d5e00496-673e-2be1-18e6-162ad07b74af} {Suffrage:Voter ID:server-22d988fe-44ff-7988-d25e-53384ef3638f Address:22d988fe-44ff-7988-d25e-53384ef3638f} {Suffrage:Voter ID:server-1af0a5fb-ff75-25a5-a673-386a8d819f02 Address:1af0a5fb-ff75-25a5-a673-386a8d819f02} {Suffrage:Voter ID:server-5b1deb0c-822b-e546-0b2c-945d89676f38 Address:5b1deb0c-822b-e546-0b2c-945d89676f38} {Suffrage:Voter ID:server-566b03f7-4507-e631-dde7-f1d7a041aa77 Address:566b03f7-4507-e631-dde7-f1d7a041aa77} {Suffrage:Voter ID:server-3dc64780-c8e3-d590-e146-de5c1a67eba7 Address:3dc64780-c8e3-d590-e146-de5c1a67eba7} {Suffrage:Voter ID:server-0d905eac-482d-e083-508f-a91a1fbc51f2 Address:0d905eac-482d-e083-508f-a91a1fbc51f2}]" +2024-07-05T13:37:19.658Z [INFO] server-5b1deb0c-822b-e546-0b2c-945d89676f38: entering follower state: follower="Node at 5b1deb0c-822b-e546-0b2c-945d89676f38 [Follower]" leader-address= leader-id= +2024-07-05T13:37:19.658Z [INFO] server-566b03f7-4507-e631-dde7-f1d7a041aa77: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5e00496-673e-2be1-18e6-162ad07b74af Address:d5e00496-673e-2be1-18e6-162ad07b74af} {Suffrage:Voter ID:server-22d988fe-44ff-7988-d25e-53384ef3638f Address:22d988fe-44ff-7988-d25e-53384ef3638f} {Suffrage:Voter ID:server-1af0a5fb-ff75-25a5-a673-386a8d819f02 Address:1af0a5fb-ff75-25a5-a673-386a8d819f02} {Suffrage:Voter ID:server-5b1deb0c-822b-e546-0b2c-945d89676f38 Address:5b1deb0c-822b-e546-0b2c-945d89676f38} {Suffrage:Voter ID:server-566b03f7-4507-e631-dde7-f1d7a041aa77 Address:566b03f7-4507-e631-dde7-f1d7a041aa77} {Suffrage:Voter ID:server-3dc64780-c8e3-d590-e146-de5c1a67eba7 Address:3dc64780-c8e3-d590-e146-de5c1a67eba7} {Suffrage:Voter ID:server-0d905eac-482d-e083-508f-a91a1fbc51f2 Address:0d905eac-482d-e083-508f-a91a1fbc51f2}]" +2024-07-05T13:37:19.659Z [INFO] server-566b03f7-4507-e631-dde7-f1d7a041aa77: entering follower state: follower="Node at 566b03f7-4507-e631-dde7-f1d7a041aa77 [Follower]" leader-address= leader-id= +2024-07-05T13:37:19.660Z [INFO] server-3dc64780-c8e3-d590-e146-de5c1a67eba7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5e00496-673e-2be1-18e6-162ad07b74af Address:d5e00496-673e-2be1-18e6-162ad07b74af} {Suffrage:Voter ID:server-22d988fe-44ff-7988-d25e-53384ef3638f Address:22d988fe-44ff-7988-d25e-53384ef3638f} {Suffrage:Voter ID:server-1af0a5fb-ff75-25a5-a673-386a8d819f02 Address:1af0a5fb-ff75-25a5-a673-386a8d819f02} {Suffrage:Voter ID:server-5b1deb0c-822b-e546-0b2c-945d89676f38 Address:5b1deb0c-822b-e546-0b2c-945d89676f38} {Suffrage:Voter ID:server-566b03f7-4507-e631-dde7-f1d7a041aa77 Address:566b03f7-4507-e631-dde7-f1d7a041aa77} {Suffrage:Voter ID:server-3dc64780-c8e3-d590-e146-de5c1a67eba7 Address:3dc64780-c8e3-d590-e146-de5c1a67eba7} {Suffrage:Voter ID:server-0d905eac-482d-e083-508f-a91a1fbc51f2 Address:0d905eac-482d-e083-508f-a91a1fbc51f2}]" +2024-07-05T13:37:19.660Z [INFO] server-3dc64780-c8e3-d590-e146-de5c1a67eba7: entering follower state: follower="Node at 3dc64780-c8e3-d590-e146-de5c1a67eba7 [Follower]" leader-address= leader-id= +2024-07-05T13:37:19.661Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5e00496-673e-2be1-18e6-162ad07b74af Address:d5e00496-673e-2be1-18e6-162ad07b74af} {Suffrage:Voter ID:server-22d988fe-44ff-7988-d25e-53384ef3638f Address:22d988fe-44ff-7988-d25e-53384ef3638f} {Suffrage:Voter ID:server-1af0a5fb-ff75-25a5-a673-386a8d819f02 Address:1af0a5fb-ff75-25a5-a673-386a8d819f02} {Suffrage:Voter ID:server-5b1deb0c-822b-e546-0b2c-945d89676f38 Address:5b1deb0c-822b-e546-0b2c-945d89676f38} {Suffrage:Voter ID:server-566b03f7-4507-e631-dde7-f1d7a041aa77 Address:566b03f7-4507-e631-dde7-f1d7a041aa77} {Suffrage:Voter ID:server-3dc64780-c8e3-d590-e146-de5c1a67eba7 Address:3dc64780-c8e3-d590-e146-de5c1a67eba7} {Suffrage:Voter ID:server-0d905eac-482d-e083-508f-a91a1fbc51f2 Address:0d905eac-482d-e083-508f-a91a1fbc51f2}]" +2024-07-05T13:37:19.661Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:19.661Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: entering follower state: follower="Node at 0d905eac-482d-e083-508f-a91a1fbc51f2 [Follower]" leader-address= leader-id= +2024-07-05T13:37:19.715Z [WARN] server-0d905eac-482d-e083-508f-a91a1fbc51f2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:19.715Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: entering candidate state: node="Node at 0d905eac-482d-e083-508f-a91a1fbc51f2 [Candidate]" term=2 +2024-07-05T13:37:19.716Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: election won: term=2 tally=4 +2024-07-05T13:37:19.716Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: entering leader state: leader="Node at 0d905eac-482d-e083-508f-a91a1fbc51f2 [Leader]" +2024-07-05T13:37:19.717Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: added peer, starting replication: peer=server-d5e00496-673e-2be1-18e6-162ad07b74af +2024-07-05T13:37:19.717Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: added peer, starting replication: peer=server-22d988fe-44ff-7988-d25e-53384ef3638f +2024-07-05T13:37:19.717Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: added peer, starting replication: peer=server-1af0a5fb-ff75-25a5-a673-386a8d819f02 +2024-07-05T13:37:19.717Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: added peer, starting replication: peer=server-5b1deb0c-822b-e546-0b2c-945d89676f38 +2024-07-05T13:37:19.717Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: added peer, starting replication: peer=server-566b03f7-4507-e631-dde7-f1d7a041aa77 +2024-07-05T13:37:19.717Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: added peer, starting replication: peer=server-3dc64780-c8e3-d590-e146-de5c1a67eba7 +2024-07-05T13:37:19.718Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: pipelining replication: peer="{Voter server-d5e00496-673e-2be1-18e6-162ad07b74af d5e00496-673e-2be1-18e6-162ad07b74af}" +2024-07-05T13:37:19.719Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: pipelining replication: peer="{Voter server-22d988fe-44ff-7988-d25e-53384ef3638f 22d988fe-44ff-7988-d25e-53384ef3638f}" +2024-07-05T13:37:19.719Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: pipelining replication: peer="{Voter server-1af0a5fb-ff75-25a5-a673-386a8d819f02 1af0a5fb-ff75-25a5-a673-386a8d819f02}" +2024-07-05T13:37:19.719Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: pipelining replication: peer="{Voter server-566b03f7-4507-e631-dde7-f1d7a041aa77 566b03f7-4507-e631-dde7-f1d7a041aa77}" +2024-07-05T13:37:19.720Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: pipelining replication: peer="{Voter server-5b1deb0c-822b-e546-0b2c-945d89676f38 5b1deb0c-822b-e546-0b2c-945d89676f38}" +2024-07-05T13:37:19.720Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: pipelining replication: peer="{Voter server-3dc64780-c8e3-d590-e146-de5c1a67eba7 3dc64780-c8e3-d590-e146-de5c1a67eba7}" +2024-07-05T13:37:19.824Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:19.717542835 +0000 UTC m=+105.713464286 (1 nodes), 55.954061ms from start of poll, 62.755896ms from cluster start. Timeout at 2024-07-05 13:37:19.823209697 +0000 UTC m=+105.819130857, 105.666571ms after stability +2024-07-05T13:37:19.824Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:19.930Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:19.825020914 +0000 UTC m=+105.820942074 (6 nodes), 10.5µs from start of poll, 170.233684ms from cluster start. Timeout at 2024-07-05 13:37:19.93009162 +0000 UTC m=+105.926012780, 105.070706ms after stability +2024-07-05T13:37:19.930Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:20.036Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:19.930441905 +0000 UTC m=+105.926362774 (1 nodes), 5.25µs from start of poll, 275.654384ms from cluster start. Timeout at 2024-07-05 13:37:20.035769857 +0000 UTC m=+106.031691017, 105.328243ms after stability +2024-07-05T13:37:20.036Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: entering candidate state: node="Node at d5e00496-673e-2be1-18e6-162ad07b74af [Candidate]" term=3 +2024-07-05T13:37:20.036Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: election won: term=3 tally=4 +2024-07-05T13:37:20.037Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: entering leader state: leader="Node at d5e00496-673e-2be1-18e6-162ad07b74af [Leader]" +2024-07-05T13:37:20.037Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: added peer, starting replication: peer=server-22d988fe-44ff-7988-d25e-53384ef3638f +2024-07-05T13:37:20.037Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:20.037Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: added peer, starting replication: peer=server-1af0a5fb-ff75-25a5-a673-386a8d819f02 +2024-07-05T13:37:20.037Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: added peer, starting replication: peer=server-5b1deb0c-822b-e546-0b2c-945d89676f38 +2024-07-05T13:37:20.037Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: added peer, starting replication: peer=server-566b03f7-4507-e631-dde7-f1d7a041aa77 +2024-07-05T13:37:20.036Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: entering follower state: follower="Node at 0d905eac-482d-e083-508f-a91a1fbc51f2 [Follower]" leader-address= leader-id= +2024-07-05T13:37:20.037Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: added peer, starting replication: peer=server-3dc64780-c8e3-d590-e146-de5c1a67eba7 +2024-07-05T13:37:20.037Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: added peer, starting replication: peer=server-0d905eac-482d-e083-508f-a91a1fbc51f2 +2024-07-05T13:37:20.037Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: aborting pipeline replication: peer="{Voter server-22d988fe-44ff-7988-d25e-53384ef3638f 22d988fe-44ff-7988-d25e-53384ef3638f}" +2024-07-05T13:37:20.037Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: aborting pipeline replication: peer="{Voter server-d5e00496-673e-2be1-18e6-162ad07b74af d5e00496-673e-2be1-18e6-162ad07b74af}" +2024-07-05T13:37:20.037Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: aborting pipeline replication: peer="{Voter server-1af0a5fb-ff75-25a5-a673-386a8d819f02 1af0a5fb-ff75-25a5-a673-386a8d819f02}" +2024-07-05T13:37:20.037Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: aborting pipeline replication: peer="{Voter server-5b1deb0c-822b-e546-0b2c-945d89676f38 5b1deb0c-822b-e546-0b2c-945d89676f38}" +2024-07-05T13:37:20.038Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: pipelining replication: peer="{Voter server-1af0a5fb-ff75-25a5-a673-386a8d819f02 1af0a5fb-ff75-25a5-a673-386a8d819f02}" +2024-07-05T13:37:20.037Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: aborting pipeline replication: peer="{Voter server-566b03f7-4507-e631-dde7-f1d7a041aa77 566b03f7-4507-e631-dde7-f1d7a041aa77}" +2024-07-05T13:37:20.038Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: pipelining replication: peer="{Voter server-22d988fe-44ff-7988-d25e-53384ef3638f 22d988fe-44ff-7988-d25e-53384ef3638f}" +2024-07-05T13:37:20.038Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: pipelining replication: peer="{Voter server-0d905eac-482d-e083-508f-a91a1fbc51f2 0d905eac-482d-e083-508f-a91a1fbc51f2}" +2024-07-05T13:37:20.037Z [INFO] server-0d905eac-482d-e083-508f-a91a1fbc51f2: aborting pipeline replication: peer="{Voter server-3dc64780-c8e3-d590-e146-de5c1a67eba7 3dc64780-c8e3-d590-e146-de5c1a67eba7}" +2024-07-05T13:37:20.038Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: pipelining replication: peer="{Voter server-5b1deb0c-822b-e546-0b2c-945d89676f38 5b1deb0c-822b-e546-0b2c-945d89676f38}" +2024-07-05T13:37:20.039Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: pipelining replication: peer="{Voter server-566b03f7-4507-e631-dde7-f1d7a041aa77 566b03f7-4507-e631-dde7-f1d7a041aa77}" +2024-07-05T13:37:20.039Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: pipelining replication: peer="{Voter server-3dc64780-c8e3-d590-e146-de5c1a67eba7 3dc64780-c8e3-d590-e146-de5c1a67eba7}" +2024-07-05T13:37:20.142Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:20.037697738 +0000 UTC m=+106.033618898 (1 nodes), 369.535µs from start of poll, 382.910508ms from cluster start. Timeout at 2024-07-05 13:37:20.14277632 +0000 UTC m=+106.138697188, 105.07829ms after stability +2024-07-05T13:37:20.143Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: aborting pipeline replication: peer="{Voter server-0d905eac-482d-e083-508f-a91a1fbc51f2 0d905eac-482d-e083-508f-a91a1fbc51f2}" +2024-07-05T13:37:20.143Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: aborting pipeline replication: peer="{Voter server-1af0a5fb-ff75-25a5-a673-386a8d819f02 1af0a5fb-ff75-25a5-a673-386a8d819f02}" +2024-07-05T13:37:20.143Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: aborting pipeline replication: peer="{Voter server-3dc64780-c8e3-d590-e146-de5c1a67eba7 3dc64780-c8e3-d590-e146-de5c1a67eba7}" +2024-07-05T13:37:20.143Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: aborting pipeline replication: peer="{Voter server-22d988fe-44ff-7988-d25e-53384ef3638f 22d988fe-44ff-7988-d25e-53384ef3638f}" +2024-07-05T13:37:20.143Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: aborting pipeline replication: peer="{Voter server-5b1deb0c-822b-e546-0b2c-945d89676f38 5b1deb0c-822b-e546-0b2c-945d89676f38}" +2024-07-05T13:37:20.143Z [INFO] server-d5e00496-673e-2be1-18e6-162ad07b74af: aborting pipeline replication: peer="{Voter server-566b03f7-4507-e631-dde7-f1d7a041aa77 566b03f7-4507-e631-dde7-f1d7a041aa77}" +2024-07-05T13:37:20.643Z [ERROR] server-d5e00496-673e-2be1-18e6-162ad07b74af: failed to heartbeat to: peer=0d905eac-482d-e083-508f-a91a1fbc51f2 backoff time=10ms error="command timed out" +--- PASS: TestRaft_LeadershipTransferWithSevenNodes (1.01s) === RUN TestRaft_LeadershipTransferToInvalidID -2024-07-05T13:31:03.373Z [INFO] server-6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c Address:6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c} {Suffrage:Voter ID:server-ffef32c1-da88-5d3c-b12b-619ffde01bec Address:ffef32c1-da88-5d3c-b12b-619ffde01bec} {Suffrage:Voter ID:server-cf225dd1-0b33-4566-be2d-c8263ea92eb2 Address:cf225dd1-0b33-4566-be2d-c8263ea92eb2}]" -2024-07-05T13:31:03.374Z [INFO] server-ffef32c1-da88-5d3c-b12b-619ffde01bec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c Address:6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c} {Suffrage:Voter ID:server-ffef32c1-da88-5d3c-b12b-619ffde01bec Address:ffef32c1-da88-5d3c-b12b-619ffde01bec} {Suffrage:Voter ID:server-cf225dd1-0b33-4566-be2d-c8263ea92eb2 Address:cf225dd1-0b33-4566-be2d-c8263ea92eb2}]" -2024-07-05T13:31:03.374Z [INFO] server-cf225dd1-0b33-4566-be2d-c8263ea92eb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c Address:6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c} {Suffrage:Voter ID:server-ffef32c1-da88-5d3c-b12b-619ffde01bec Address:ffef32c1-da88-5d3c-b12b-619ffde01bec} {Suffrage:Voter ID:server-cf225dd1-0b33-4566-be2d-c8263ea92eb2 Address:cf225dd1-0b33-4566-be2d-c8263ea92eb2}]" -2024-07-05T13:31:03.374Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:03.374Z [INFO] server-6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c: entering follower state: follower="Node at 6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c [Follower]" leader-address= leader-id= -2024-07-05T13:31:03.374Z [INFO] server-ffef32c1-da88-5d3c-b12b-619ffde01bec: entering follower state: follower="Node at ffef32c1-da88-5d3c-b12b-619ffde01bec [Follower]" leader-address= leader-id= -2024-07-05T13:31:03.374Z [INFO] server-cf225dd1-0b33-4566-be2d-c8263ea92eb2: entering follower state: follower="Node at cf225dd1-0b33-4566-be2d-c8263ea92eb2 [Follower]" leader-address= leader-id= -2024-07-05T13:31:03.455Z [WARN] server-6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:03.455Z [INFO] server-6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c: entering candidate state: node="Node at 6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c [Candidate]" term=2 -2024-07-05T13:31:03.455Z [INFO] server-6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c: election won: term=2 tally=2 -2024-07-05T13:31:03.455Z [INFO] server-6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c: entering leader state: leader="Node at 6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c [Leader]" -2024-07-05T13:31:03.455Z [INFO] server-6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c: added peer, starting replication: peer=server-ffef32c1-da88-5d3c-b12b-619ffde01bec -2024-07-05T13:31:03.455Z [INFO] server-6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c: added peer, starting replication: peer=server-cf225dd1-0b33-4566-be2d-c8263ea92eb2 -2024-07-05T13:31:03.456Z [INFO] server-6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c: pipelining replication: peer="{Voter server-ffef32c1-da88-5d3c-b12b-619ffde01bec ffef32c1-da88-5d3c-b12b-619ffde01bec}" -2024-07-05T13:31:03.456Z [INFO] server-6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c: pipelining replication: peer="{Voter server-cf225dd1-0b33-4566-be2d-c8263ea92eb2 cf225dd1-0b33-4566-be2d-c8263ea92eb2}" -2024-07-05T13:31:03.562Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:03.456491811 +0000 UTC m=+104.226402526 (1 nodes), 82.084481ms from start of poll, 82.870213ms from cluster start. Timeout at 2024-07-05 13:31:03.562604368 +0000 UTC m=+104.332515043, 106.112517ms after stability -2024-07-05T13:31:03.563Z [INFO] server-6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c: aborting pipeline replication: peer="{Voter server-cf225dd1-0b33-4566-be2d-c8263ea92eb2 cf225dd1-0b33-4566-be2d-c8263ea92eb2}" -2024-07-05T13:31:03.563Z [INFO] server-6cb53ae6-d6d9-1029-0e78-d4ea45d17b9c: aborting pipeline replication: peer="{Voter server-ffef32c1-da88-5d3c-b12b-619ffde01bec ffef32c1-da88-5d3c-b12b-619ffde01bec}" +2024-07-05T13:37:20.656Z [INFO] server-0bc4258e-263e-df75-f288-da6c6fcfa689: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bc4258e-263e-df75-f288-da6c6fcfa689 Address:0bc4258e-263e-df75-f288-da6c6fcfa689} {Suffrage:Voter ID:server-76b9ced4-2aeb-0eb3-5064-f14740ad9714 Address:76b9ced4-2aeb-0eb3-5064-f14740ad9714} {Suffrage:Voter ID:server-e0942f7b-1144-b0b7-4c48-f3a37bca0a27 Address:e0942f7b-1144-b0b7-4c48-f3a37bca0a27}]" +2024-07-05T13:37:20.656Z [INFO] server-0bc4258e-263e-df75-f288-da6c6fcfa689: entering follower state: follower="Node at 0bc4258e-263e-df75-f288-da6c6fcfa689 [Follower]" leader-address= leader-id= +2024-07-05T13:37:20.656Z [INFO] server-76b9ced4-2aeb-0eb3-5064-f14740ad9714: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bc4258e-263e-df75-f288-da6c6fcfa689 Address:0bc4258e-263e-df75-f288-da6c6fcfa689} {Suffrage:Voter ID:server-76b9ced4-2aeb-0eb3-5064-f14740ad9714 Address:76b9ced4-2aeb-0eb3-5064-f14740ad9714} {Suffrage:Voter ID:server-e0942f7b-1144-b0b7-4c48-f3a37bca0a27 Address:e0942f7b-1144-b0b7-4c48-f3a37bca0a27}]" +2024-07-05T13:37:20.657Z [INFO] server-e0942f7b-1144-b0b7-4c48-f3a37bca0a27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bc4258e-263e-df75-f288-da6c6fcfa689 Address:0bc4258e-263e-df75-f288-da6c6fcfa689} {Suffrage:Voter ID:server-76b9ced4-2aeb-0eb3-5064-f14740ad9714 Address:76b9ced4-2aeb-0eb3-5064-f14740ad9714} {Suffrage:Voter ID:server-e0942f7b-1144-b0b7-4c48-f3a37bca0a27 Address:e0942f7b-1144-b0b7-4c48-f3a37bca0a27}]" +2024-07-05T13:37:20.657Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:20.657Z [INFO] server-76b9ced4-2aeb-0eb3-5064-f14740ad9714: entering follower state: follower="Node at 76b9ced4-2aeb-0eb3-5064-f14740ad9714 [Follower]" leader-address= leader-id= +2024-07-05T13:37:20.657Z [INFO] server-e0942f7b-1144-b0b7-4c48-f3a37bca0a27: entering follower state: follower="Node at e0942f7b-1144-b0b7-4c48-f3a37bca0a27 [Follower]" leader-address= leader-id= +2024-07-05T13:37:20.726Z [WARN] server-e0942f7b-1144-b0b7-4c48-f3a37bca0a27: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:20.726Z [INFO] server-e0942f7b-1144-b0b7-4c48-f3a37bca0a27: entering candidate state: node="Node at e0942f7b-1144-b0b7-4c48-f3a37bca0a27 [Candidate]" term=2 +2024-07-05T13:37:20.727Z [INFO] server-e0942f7b-1144-b0b7-4c48-f3a37bca0a27: election won: term=2 tally=2 +2024-07-05T13:37:20.727Z [INFO] server-e0942f7b-1144-b0b7-4c48-f3a37bca0a27: entering leader state: leader="Node at e0942f7b-1144-b0b7-4c48-f3a37bca0a27 [Leader]" +2024-07-05T13:37:20.727Z [INFO] server-e0942f7b-1144-b0b7-4c48-f3a37bca0a27: added peer, starting replication: peer=server-0bc4258e-263e-df75-f288-da6c6fcfa689 +2024-07-05T13:37:20.727Z [INFO] server-e0942f7b-1144-b0b7-4c48-f3a37bca0a27: added peer, starting replication: peer=server-76b9ced4-2aeb-0eb3-5064-f14740ad9714 +2024-07-05T13:37:20.728Z [INFO] server-e0942f7b-1144-b0b7-4c48-f3a37bca0a27: pipelining replication: peer="{Voter server-76b9ced4-2aeb-0eb3-5064-f14740ad9714 76b9ced4-2aeb-0eb3-5064-f14740ad9714}" +2024-07-05T13:37:20.728Z [INFO] server-e0942f7b-1144-b0b7-4c48-f3a37bca0a27: pipelining replication: peer="{Voter server-0bc4258e-263e-df75-f288-da6c6fcfa689 0bc4258e-263e-df75-f288-da6c6fcfa689}" +2024-07-05T13:37:20.832Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:20.727379978 +0000 UTC m=+106.723301138 (1 nodes), 69.83335ms from start of poll, 71.791856ms from cluster start. Timeout at 2024-07-05 13:37:20.832499683 +0000 UTC m=+106.828421135, 105.119997ms after stability +2024-07-05T13:37:20.833Z [INFO] server-e0942f7b-1144-b0b7-4c48-f3a37bca0a27: aborting pipeline replication: peer="{Voter server-76b9ced4-2aeb-0eb3-5064-f14740ad9714 76b9ced4-2aeb-0eb3-5064-f14740ad9714}" +2024-07-05T13:37:20.833Z [INFO] server-e0942f7b-1144-b0b7-4c48-f3a37bca0a27: aborting pipeline replication: peer="{Voter server-0bc4258e-263e-df75-f288-da6c6fcfa689 0bc4258e-263e-df75-f288-da6c6fcfa689}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.19s) === RUN TestRaft_LeadershipTransferToInvalidAddress -2024-07-05T13:31:03.565Z [INFO] server-084fc548-983b-110c-65fa-e89b2879a06b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-084fc548-983b-110c-65fa-e89b2879a06b Address:084fc548-983b-110c-65fa-e89b2879a06b} {Suffrage:Voter ID:server-61119211-a57f-5a61-c3ba-35d1b0c77159 Address:61119211-a57f-5a61-c3ba-35d1b0c77159} {Suffrage:Voter ID:server-3659cb41-4eab-0591-4b67-ba80a7f92aba Address:3659cb41-4eab-0591-4b67-ba80a7f92aba}]" -2024-07-05T13:31:03.566Z [INFO] server-084fc548-983b-110c-65fa-e89b2879a06b: entering follower state: follower="Node at 084fc548-983b-110c-65fa-e89b2879a06b [Follower]" leader-address= leader-id= -2024-07-05T13:31:03.566Z [INFO] server-61119211-a57f-5a61-c3ba-35d1b0c77159: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-084fc548-983b-110c-65fa-e89b2879a06b Address:084fc548-983b-110c-65fa-e89b2879a06b} {Suffrage:Voter ID:server-61119211-a57f-5a61-c3ba-35d1b0c77159 Address:61119211-a57f-5a61-c3ba-35d1b0c77159} {Suffrage:Voter ID:server-3659cb41-4eab-0591-4b67-ba80a7f92aba Address:3659cb41-4eab-0591-4b67-ba80a7f92aba}]" -2024-07-05T13:31:03.566Z [INFO] server-61119211-a57f-5a61-c3ba-35d1b0c77159: entering follower state: follower="Node at 61119211-a57f-5a61-c3ba-35d1b0c77159 [Follower]" leader-address= leader-id= -2024-07-05T13:31:03.566Z [INFO] server-3659cb41-4eab-0591-4b67-ba80a7f92aba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-084fc548-983b-110c-65fa-e89b2879a06b Address:084fc548-983b-110c-65fa-e89b2879a06b} {Suffrage:Voter ID:server-61119211-a57f-5a61-c3ba-35d1b0c77159 Address:61119211-a57f-5a61-c3ba-35d1b0c77159} {Suffrage:Voter ID:server-3659cb41-4eab-0591-4b67-ba80a7f92aba Address:3659cb41-4eab-0591-4b67-ba80a7f92aba}]" -2024-07-05T13:31:03.566Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:31:03.566Z [INFO] server-3659cb41-4eab-0591-4b67-ba80a7f92aba: entering follower state: follower="Node at 3659cb41-4eab-0591-4b67-ba80a7f92aba [Follower]" leader-address= leader-id= -2024-07-05T13:31:03.624Z [WARN] server-084fc548-983b-110c-65fa-e89b2879a06b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:03.624Z [INFO] server-084fc548-983b-110c-65fa-e89b2879a06b: entering candidate state: node="Node at 084fc548-983b-110c-65fa-e89b2879a06b [Candidate]" term=2 -2024-07-05T13:31:03.625Z [INFO] server-084fc548-983b-110c-65fa-e89b2879a06b: election won: term=2 tally=2 -2024-07-05T13:31:03.625Z [INFO] server-084fc548-983b-110c-65fa-e89b2879a06b: entering leader state: leader="Node at 084fc548-983b-110c-65fa-e89b2879a06b [Leader]" -2024-07-05T13:31:03.625Z [INFO] server-084fc548-983b-110c-65fa-e89b2879a06b: added peer, starting replication: peer=server-61119211-a57f-5a61-c3ba-35d1b0c77159 -2024-07-05T13:31:03.625Z [INFO] server-084fc548-983b-110c-65fa-e89b2879a06b: added peer, starting replication: peer=server-3659cb41-4eab-0591-4b67-ba80a7f92aba -2024-07-05T13:31:03.625Z [INFO] server-084fc548-983b-110c-65fa-e89b2879a06b: pipelining replication: peer="{Voter server-3659cb41-4eab-0591-4b67-ba80a7f92aba 3659cb41-4eab-0591-4b67-ba80a7f92aba}" -2024-07-05T13:31:03.625Z [INFO] server-084fc548-983b-110c-65fa-e89b2879a06b: pipelining replication: peer="{Voter server-61119211-a57f-5a61-c3ba-35d1b0c77159 61119211-a57f-5a61-c3ba-35d1b0c77159}" -2024-07-05T13:31:03.731Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:31:03.625581838 +0000 UTC m=+104.395492513 (2 nodes), 58.930172ms from start of poll, 59.893179ms from cluster start. Timeout at 2024-07-05 13:31:03.731522441 +0000 UTC m=+104.501433156, 105.940643ms after stability -2024-07-05T13:31:03.731Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:03.837Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:03.731746993 +0000 UTC m=+104.501657668 (1 nodes), 1.6µs from start of poll, 166.058334ms from cluster start. Timeout at 2024-07-05 13:31:03.837185974 +0000 UTC m=+104.607096609, 105.438941ms after stability -2024-07-05T13:31:03.837Z [INFO] server-084fc548-983b-110c-65fa-e89b2879a06b: aborting pipeline replication: peer="{Voter server-3659cb41-4eab-0591-4b67-ba80a7f92aba 3659cb41-4eab-0591-4b67-ba80a7f92aba}" -2024-07-05T13:31:03.837Z [INFO] server-084fc548-983b-110c-65fa-e89b2879a06b: aborting pipeline replication: peer="{Voter server-61119211-a57f-5a61-c3ba-35d1b0c77159 61119211-a57f-5a61-c3ba-35d1b0c77159}" ---- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.27s) +2024-07-05T13:37:20.844Z [INFO] server-c1f5219d-83cb-f277-bc69-a95502606f7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1f5219d-83cb-f277-bc69-a95502606f7a Address:c1f5219d-83cb-f277-bc69-a95502606f7a} {Suffrage:Voter ID:server-070b4d6f-0692-073a-6c21-367f1878d8d5 Address:070b4d6f-0692-073a-6c21-367f1878d8d5} {Suffrage:Voter ID:server-a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39 Address:a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39}]" +2024-07-05T13:37:20.844Z [INFO] server-c1f5219d-83cb-f277-bc69-a95502606f7a: entering follower state: follower="Node at c1f5219d-83cb-f277-bc69-a95502606f7a [Follower]" leader-address= leader-id= +2024-07-05T13:37:20.844Z [INFO] server-070b4d6f-0692-073a-6c21-367f1878d8d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1f5219d-83cb-f277-bc69-a95502606f7a Address:c1f5219d-83cb-f277-bc69-a95502606f7a} {Suffrage:Voter ID:server-070b4d6f-0692-073a-6c21-367f1878d8d5 Address:070b4d6f-0692-073a-6c21-367f1878d8d5} {Suffrage:Voter ID:server-a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39 Address:a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39}]" +2024-07-05T13:37:20.845Z [INFO] server-070b4d6f-0692-073a-6c21-367f1878d8d5: entering follower state: follower="Node at 070b4d6f-0692-073a-6c21-367f1878d8d5 [Follower]" leader-address= leader-id= +2024-07-05T13:37:20.845Z [INFO] server-a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1f5219d-83cb-f277-bc69-a95502606f7a Address:c1f5219d-83cb-f277-bc69-a95502606f7a} {Suffrage:Voter ID:server-070b4d6f-0692-073a-6c21-367f1878d8d5 Address:070b4d6f-0692-073a-6c21-367f1878d8d5} {Suffrage:Voter ID:server-a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39 Address:a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39}]" +2024-07-05T13:37:20.845Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:20.845Z [INFO] server-a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39: entering follower state: follower="Node at a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39 [Follower]" leader-address= leader-id= +2024-07-05T13:37:20.909Z [WARN] server-a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:20.909Z [INFO] server-a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39: entering candidate state: node="Node at a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39 [Candidate]" term=2 +2024-07-05T13:37:20.910Z [INFO] server-a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39: election won: term=2 tally=2 +2024-07-05T13:37:20.910Z [INFO] server-a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39: entering leader state: leader="Node at a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39 [Leader]" +2024-07-05T13:37:20.911Z [INFO] server-a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39: added peer, starting replication: peer=server-c1f5219d-83cb-f277-bc69-a95502606f7a +2024-07-05T13:37:20.911Z [INFO] server-a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39: added peer, starting replication: peer=server-070b4d6f-0692-073a-6c21-367f1878d8d5 +2024-07-05T13:37:20.912Z [INFO] server-a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39: pipelining replication: peer="{Voter server-070b4d6f-0692-073a-6c21-367f1878d8d5 070b4d6f-0692-073a-6c21-367f1878d8d5}" +2024-07-05T13:37:20.913Z [WARN] server-c1f5219d-83cb-f277-bc69-a95502606f7a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:20.913Z [INFO] server-c1f5219d-83cb-f277-bc69-a95502606f7a: entering candidate state: node="Node at c1f5219d-83cb-f277-bc69-a95502606f7a [Candidate]" term=2 +2024-07-05T13:37:20.913Z [INFO] server-c1f5219d-83cb-f277-bc69-a95502606f7a: duplicate requestVote for same term: term=2 +2024-07-05T13:37:20.913Z [INFO] server-c1f5219d-83cb-f277-bc69-a95502606f7a: entering follower state: follower="Node at c1f5219d-83cb-f277-bc69-a95502606f7a [Follower]" leader-address=a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39 leader-id=server-a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39 +2024-07-05T13:37:20.914Z [INFO] server-a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39: pipelining replication: peer="{Voter server-c1f5219d-83cb-f277-bc69-a95502606f7a c1f5219d-83cb-f277-bc69-a95502606f7a}" +2024-07-05T13:37:20.914Z [WARN] server-070b4d6f-0692-073a-6c21-367f1878d8d5: rejecting vote request since we have a leader: from=c1f5219d-83cb-f277-bc69-a95502606f7a leader=a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39 leader-id=server-a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39 +2024-07-05T13:37:20.914Z [WARN] server-a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39: rejecting vote request since we have a leader: from=c1f5219d-83cb-f277-bc69-a95502606f7a leader=a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39 leader-id=server-a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39 +2024-07-05T13:37:21.020Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:20.915834036 +0000 UTC m=+106.911755196 (2 nodes), 70.069595ms from start of poll, 72.415719ms from cluster start. Timeout at 2024-07-05 13:37:21.02085341 +0000 UTC m=+107.016774570, 105.019374ms after stability +2024-07-05T13:37:21.021Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:21.127Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:21.021238987 +0000 UTC m=+107.017160147 (1 nodes), 4.375µs from start of poll, 177.82067ms from cluster start. Timeout at 2024-07-05 13:37:21.127747875 +0000 UTC m=+107.123668743, 106.508596ms after stability +2024-07-05T13:37:21.128Z [INFO] server-a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39: aborting pipeline replication: peer="{Voter server-070b4d6f-0692-073a-6c21-367f1878d8d5 070b4d6f-0692-073a-6c21-367f1878d8d5}" +2024-07-05T13:37:21.128Z [INFO] server-a20bc68c-3f3e-11fb-75e4-c5d3e0bc4a39: aborting pipeline replication: peer="{Voter server-c1f5219d-83cb-f277-bc69-a95502606f7a c1f5219d-83cb-f277-bc69-a95502606f7a}" +--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.29s) === RUN TestRaft_LeadershipTransferToBehindServer -2024-07-05T13:31:03.839Z [INFO] server-e775c984-433f-97e2-b070-11d795d8fd70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e775c984-433f-97e2-b070-11d795d8fd70 Address:e775c984-433f-97e2-b070-11d795d8fd70} {Suffrage:Voter ID:server-6ec898d7-14f7-a227-359e-16d0f652a778 Address:6ec898d7-14f7-a227-359e-16d0f652a778} {Suffrage:Voter ID:server-d19af6bb-0ce8-540b-74d3-a0d115ce81d1 Address:d19af6bb-0ce8-540b-74d3-a0d115ce81d1}]" -2024-07-05T13:31:03.840Z [INFO] server-e775c984-433f-97e2-b070-11d795d8fd70: entering follower state: follower="Node at e775c984-433f-97e2-b070-11d795d8fd70 [Follower]" leader-address= leader-id= -2024-07-05T13:31:03.840Z [INFO] server-6ec898d7-14f7-a227-359e-16d0f652a778: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e775c984-433f-97e2-b070-11d795d8fd70 Address:e775c984-433f-97e2-b070-11d795d8fd70} {Suffrage:Voter ID:server-6ec898d7-14f7-a227-359e-16d0f652a778 Address:6ec898d7-14f7-a227-359e-16d0f652a778} {Suffrage:Voter ID:server-d19af6bb-0ce8-540b-74d3-a0d115ce81d1 Address:d19af6bb-0ce8-540b-74d3-a0d115ce81d1}]" -2024-07-05T13:31:03.840Z [INFO] server-d19af6bb-0ce8-540b-74d3-a0d115ce81d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e775c984-433f-97e2-b070-11d795d8fd70 Address:e775c984-433f-97e2-b070-11d795d8fd70} {Suffrage:Voter ID:server-6ec898d7-14f7-a227-359e-16d0f652a778 Address:6ec898d7-14f7-a227-359e-16d0f652a778} {Suffrage:Voter ID:server-d19af6bb-0ce8-540b-74d3-a0d115ce81d1 Address:d19af6bb-0ce8-540b-74d3-a0d115ce81d1}]" -2024-07-05T13:31:03.840Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:03.840Z [INFO] server-6ec898d7-14f7-a227-359e-16d0f652a778: entering follower state: follower="Node at 6ec898d7-14f7-a227-359e-16d0f652a778 [Follower]" leader-address= leader-id= -2024-07-05T13:31:03.840Z [INFO] server-d19af6bb-0ce8-540b-74d3-a0d115ce81d1: entering follower state: follower="Node at d19af6bb-0ce8-540b-74d3-a0d115ce81d1 [Follower]" leader-address= leader-id= -2024-07-05T13:31:03.895Z [WARN] server-e775c984-433f-97e2-b070-11d795d8fd70: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:03.895Z [INFO] server-e775c984-433f-97e2-b070-11d795d8fd70: entering candidate state: node="Node at e775c984-433f-97e2-b070-11d795d8fd70 [Candidate]" term=2 -2024-07-05T13:31:03.895Z [INFO] server-e775c984-433f-97e2-b070-11d795d8fd70: election won: term=2 tally=2 -2024-07-05T13:31:03.895Z [INFO] server-e775c984-433f-97e2-b070-11d795d8fd70: entering leader state: leader="Node at e775c984-433f-97e2-b070-11d795d8fd70 [Leader]" -2024-07-05T13:31:03.895Z [INFO] server-e775c984-433f-97e2-b070-11d795d8fd70: added peer, starting replication: peer=server-6ec898d7-14f7-a227-359e-16d0f652a778 -2024-07-05T13:31:03.895Z [INFO] server-e775c984-433f-97e2-b070-11d795d8fd70: added peer, starting replication: peer=server-d19af6bb-0ce8-540b-74d3-a0d115ce81d1 -2024-07-05T13:31:03.896Z [INFO] server-e775c984-433f-97e2-b070-11d795d8fd70: pipelining replication: peer="{Voter server-d19af6bb-0ce8-540b-74d3-a0d115ce81d1 d19af6bb-0ce8-540b-74d3-a0d115ce81d1}" -2024-07-05T13:31:03.896Z [INFO] server-e775c984-433f-97e2-b070-11d795d8fd70: pipelining replication: peer="{Voter server-6ec898d7-14f7-a227-359e-16d0f652a778 6ec898d7-14f7-a227-359e-16d0f652a778}" -2024-07-05T13:31:04.008Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:03.896753724 +0000 UTC m=+104.666664399 (1 nodes), 56.053753ms from start of poll, 57.089637ms from cluster start. Timeout at 2024-07-05 13:31:04.004187115 +0000 UTC m=+104.774097790, 107.433391ms after stability -2024-07-05T13:31:04.008Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:31:04.114Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:31:04.008106737 +0000 UTC m=+104.778017412 (2 nodes), 2.36µs from start of poll, 168.44265ms from cluster start. Timeout at 2024-07-05 13:31:04.11407286 +0000 UTC m=+104.883983535, 105.966123ms after stability -2024-07-05T13:31:04.260Z [INFO] server-6ec898d7-14f7-a227-359e-16d0f652a778: entering candidate state: node="Node at 6ec898d7-14f7-a227-359e-16d0f652a778 [Candidate]" term=3 -2024-07-05T13:31:04.260Z [INFO] server-6ec898d7-14f7-a227-359e-16d0f652a778: election won: term=3 tally=2 -2024-07-05T13:31:04.260Z [INFO] server-6ec898d7-14f7-a227-359e-16d0f652a778: entering leader state: leader="Node at 6ec898d7-14f7-a227-359e-16d0f652a778 [Leader]" -2024-07-05T13:31:04.260Z [INFO] server-6ec898d7-14f7-a227-359e-16d0f652a778: added peer, starting replication: peer=server-e775c984-433f-97e2-b070-11d795d8fd70 -2024-07-05T13:31:04.260Z [INFO] server-6ec898d7-14f7-a227-359e-16d0f652a778: added peer, starting replication: peer=server-d19af6bb-0ce8-540b-74d3-a0d115ce81d1 -2024-07-05T13:31:04.261Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:04.261Z [INFO] server-e775c984-433f-97e2-b070-11d795d8fd70: entering follower state: follower="Node at e775c984-433f-97e2-b070-11d795d8fd70 [Follower]" leader-address= leader-id= -2024-07-05T13:31:04.261Z [INFO] server-e775c984-433f-97e2-b070-11d795d8fd70: aborting pipeline replication: peer="{Voter server-d19af6bb-0ce8-540b-74d3-a0d115ce81d1 d19af6bb-0ce8-540b-74d3-a0d115ce81d1}" -2024-07-05T13:31:04.261Z [INFO] server-6ec898d7-14f7-a227-359e-16d0f652a778: pipelining replication: peer="{Voter server-e775c984-433f-97e2-b070-11d795d8fd70 e775c984-433f-97e2-b070-11d795d8fd70}" -2024-07-05T13:31:04.261Z [INFO] server-6ec898d7-14f7-a227-359e-16d0f652a778: pipelining replication: peer="{Voter server-d19af6bb-0ce8-540b-74d3-a0d115ce81d1 d19af6bb-0ce8-540b-74d3-a0d115ce81d1}" -2024-07-05T13:31:04.261Z [INFO] server-e775c984-433f-97e2-b070-11d795d8fd70: aborting pipeline replication: peer="{Voter server-6ec898d7-14f7-a227-359e-16d0f652a778 6ec898d7-14f7-a227-359e-16d0f652a778}" -2024-07-05T13:31:04.366Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:04.261736359 +0000 UTC m=+105.031647034 (1 nodes), 686.136µs from start of poll, 422.072272ms from cluster start. Timeout at 2024-07-05 13:31:04.366850391 +0000 UTC m=+105.136761066, 105.114032ms after stability -2024-07-05T13:31:04.367Z [INFO] server-6ec898d7-14f7-a227-359e-16d0f652a778: aborting pipeline replication: peer="{Voter server-d19af6bb-0ce8-540b-74d3-a0d115ce81d1 d19af6bb-0ce8-540b-74d3-a0d115ce81d1}" -2024-07-05T13:31:04.367Z [INFO] server-6ec898d7-14f7-a227-359e-16d0f652a778: aborting pipeline replication: peer="{Voter server-e775c984-433f-97e2-b070-11d795d8fd70 e775c984-433f-97e2-b070-11d795d8fd70}" ---- PASS: TestRaft_LeadershipTransferToBehindServer (0.53s) +2024-07-05T13:37:21.137Z [INFO] server-59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c Address:59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c} {Suffrage:Voter ID:server-7ff33912-3c60-8ccc-3e1d-fafcaac6306a Address:7ff33912-3c60-8ccc-3e1d-fafcaac6306a} {Suffrage:Voter ID:server-b63d6b47-6504-657b-839d-5ecb85a72637 Address:b63d6b47-6504-657b-839d-5ecb85a72637}]" +2024-07-05T13:37:21.137Z [INFO] server-59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c: entering follower state: follower="Node at 59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c [Follower]" leader-address= leader-id= +2024-07-05T13:37:21.138Z [INFO] server-7ff33912-3c60-8ccc-3e1d-fafcaac6306a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c Address:59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c} {Suffrage:Voter ID:server-7ff33912-3c60-8ccc-3e1d-fafcaac6306a Address:7ff33912-3c60-8ccc-3e1d-fafcaac6306a} {Suffrage:Voter ID:server-b63d6b47-6504-657b-839d-5ecb85a72637 Address:b63d6b47-6504-657b-839d-5ecb85a72637}]" +2024-07-05T13:37:21.138Z [INFO] server-7ff33912-3c60-8ccc-3e1d-fafcaac6306a: entering follower state: follower="Node at 7ff33912-3c60-8ccc-3e1d-fafcaac6306a [Follower]" leader-address= leader-id= +2024-07-05T13:37:21.138Z [INFO] server-b63d6b47-6504-657b-839d-5ecb85a72637: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c Address:59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c} {Suffrage:Voter ID:server-7ff33912-3c60-8ccc-3e1d-fafcaac6306a Address:7ff33912-3c60-8ccc-3e1d-fafcaac6306a} {Suffrage:Voter ID:server-b63d6b47-6504-657b-839d-5ecb85a72637 Address:b63d6b47-6504-657b-839d-5ecb85a72637}]" +2024-07-05T13:37:21.138Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:21.138Z [INFO] server-b63d6b47-6504-657b-839d-5ecb85a72637: entering follower state: follower="Node at b63d6b47-6504-657b-839d-5ecb85a72637 [Follower]" leader-address= leader-id= +2024-07-05T13:37:21.203Z [WARN] server-59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:21.203Z [INFO] server-59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c: entering candidate state: node="Node at 59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c [Candidate]" term=2 +2024-07-05T13:37:21.215Z [INFO] server-59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c: election won: term=2 tally=2 +2024-07-05T13:37:21.215Z [INFO] server-59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c: entering leader state: leader="Node at 59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c [Leader]" +2024-07-05T13:37:21.215Z [INFO] server-59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c: added peer, starting replication: peer=server-7ff33912-3c60-8ccc-3e1d-fafcaac6306a +2024-07-05T13:37:21.216Z [INFO] server-59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c: added peer, starting replication: peer=server-b63d6b47-6504-657b-839d-5ecb85a72637 +2024-07-05T13:37:21.217Z [INFO] server-59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c: pipelining replication: peer="{Voter server-7ff33912-3c60-8ccc-3e1d-fafcaac6306a 7ff33912-3c60-8ccc-3e1d-fafcaac6306a}" +2024-07-05T13:37:21.218Z [INFO] server-59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c: pipelining replication: peer="{Voter server-b63d6b47-6504-657b-839d-5ecb85a72637 b63d6b47-6504-657b-839d-5ecb85a72637}" +2024-07-05T13:37:21.322Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:21.216765541 +0000 UTC m=+107.212686410 (1 nodes), 77.906245ms from start of poll, 79.96304ms from cluster start. Timeout at 2024-07-05 13:37:21.32177879 +0000 UTC m=+107.317700242, 105.013832ms after stability +2024-07-05T13:37:21.322Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:21.427Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:21.322257407 +0000 UTC m=+107.318178858 (2 nodes), 8.458µs from start of poll, 185.455488ms from cluster start. Timeout at 2024-07-05 13:37:21.427381195 +0000 UTC m=+107.423302064, 105.123206ms after stability +2024-07-05T13:37:21.624Z [INFO] server-7ff33912-3c60-8ccc-3e1d-fafcaac6306a: entering candidate state: node="Node at 7ff33912-3c60-8ccc-3e1d-fafcaac6306a [Candidate]" term=3 +2024-07-05T13:37:21.624Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:21.624Z [INFO] server-7ff33912-3c60-8ccc-3e1d-fafcaac6306a: election won: term=3 tally=2 +2024-07-05T13:37:21.624Z [INFO] server-7ff33912-3c60-8ccc-3e1d-fafcaac6306a: entering leader state: leader="Node at 7ff33912-3c60-8ccc-3e1d-fafcaac6306a [Leader]" +2024-07-05T13:37:21.625Z [INFO] server-7ff33912-3c60-8ccc-3e1d-fafcaac6306a: added peer, starting replication: peer=server-59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c +2024-07-05T13:37:21.625Z [INFO] server-59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c: entering follower state: follower="Node at 59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c [Follower]" leader-address= leader-id= +2024-07-05T13:37:21.625Z [INFO] server-7ff33912-3c60-8ccc-3e1d-fafcaac6306a: added peer, starting replication: peer=server-b63d6b47-6504-657b-839d-5ecb85a72637 +2024-07-05T13:37:21.625Z [INFO] server-59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c: aborting pipeline replication: peer="{Voter server-b63d6b47-6504-657b-839d-5ecb85a72637 b63d6b47-6504-657b-839d-5ecb85a72637}" +2024-07-05T13:37:21.625Z [INFO] server-59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c: aborting pipeline replication: peer="{Voter server-7ff33912-3c60-8ccc-3e1d-fafcaac6306a 7ff33912-3c60-8ccc-3e1d-fafcaac6306a}" +2024-07-05T13:37:21.625Z [INFO] server-7ff33912-3c60-8ccc-3e1d-fafcaac6306a: pipelining replication: peer="{Voter server-b63d6b47-6504-657b-839d-5ecb85a72637 b63d6b47-6504-657b-839d-5ecb85a72637}" +2024-07-05T13:37:21.625Z [INFO] server-7ff33912-3c60-8ccc-3e1d-fafcaac6306a: pipelining replication: peer="{Voter server-59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c 59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c}" +2024-07-05T13:37:21.730Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:21.625208374 +0000 UTC m=+107.621128951 (1 nodes), 382.66µs from start of poll, 488.405581ms from cluster start. Timeout at 2024-07-05 13:37:21.730644533 +0000 UTC m=+107.726565401, 105.43645ms after stability +2024-07-05T13:37:21.730Z [INFO] server-7ff33912-3c60-8ccc-3e1d-fafcaac6306a: aborting pipeline replication: peer="{Voter server-b63d6b47-6504-657b-839d-5ecb85a72637 b63d6b47-6504-657b-839d-5ecb85a72637}" +2024-07-05T13:37:21.730Z [INFO] server-7ff33912-3c60-8ccc-3e1d-fafcaac6306a: aborting pipeline replication: peer="{Voter server-59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c 59a085ae-dd4d-19b6-7cc0-f46d2e9a5d1c}" +--- PASS: TestRaft_LeadershipTransferToBehindServer (0.60s) === RUN TestRaft_LeadershipTransferToItself -2024-07-05T13:31:04.370Z [INFO] server-fbd8c553-0946-8822-1c7d-f33a6a289ead: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbd8c553-0946-8822-1c7d-f33a6a289ead Address:fbd8c553-0946-8822-1c7d-f33a6a289ead} {Suffrage:Voter ID:server-007edb95-fb76-8331-adce-852584231e8c Address:007edb95-fb76-8331-adce-852584231e8c} {Suffrage:Voter ID:server-3e681a2d-3c48-26ba-82ba-c3608f676b7f Address:3e681a2d-3c48-26ba-82ba-c3608f676b7f}]" -2024-07-05T13:31:04.370Z [INFO] server-007edb95-fb76-8331-adce-852584231e8c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbd8c553-0946-8822-1c7d-f33a6a289ead Address:fbd8c553-0946-8822-1c7d-f33a6a289ead} {Suffrage:Voter ID:server-007edb95-fb76-8331-adce-852584231e8c Address:007edb95-fb76-8331-adce-852584231e8c} {Suffrage:Voter ID:server-3e681a2d-3c48-26ba-82ba-c3608f676b7f Address:3e681a2d-3c48-26ba-82ba-c3608f676b7f}]" -2024-07-05T13:31:04.371Z [INFO] server-3e681a2d-3c48-26ba-82ba-c3608f676b7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbd8c553-0946-8822-1c7d-f33a6a289ead Address:fbd8c553-0946-8822-1c7d-f33a6a289ead} {Suffrage:Voter ID:server-007edb95-fb76-8331-adce-852584231e8c Address:007edb95-fb76-8331-adce-852584231e8c} {Suffrage:Voter ID:server-3e681a2d-3c48-26ba-82ba-c3608f676b7f Address:3e681a2d-3c48-26ba-82ba-c3608f676b7f}]" -2024-07-05T13:31:04.371Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:04.371Z [INFO] server-fbd8c553-0946-8822-1c7d-f33a6a289ead: entering follower state: follower="Node at fbd8c553-0946-8822-1c7d-f33a6a289ead [Follower]" leader-address= leader-id= -2024-07-05T13:31:04.371Z [INFO] server-007edb95-fb76-8331-adce-852584231e8c: entering follower state: follower="Node at 007edb95-fb76-8331-adce-852584231e8c [Follower]" leader-address= leader-id= -2024-07-05T13:31:04.371Z [INFO] server-3e681a2d-3c48-26ba-82ba-c3608f676b7f: entering follower state: follower="Node at 3e681a2d-3c48-26ba-82ba-c3608f676b7f [Follower]" leader-address= leader-id= -2024-07-05T13:31:04.437Z [WARN] server-3e681a2d-3c48-26ba-82ba-c3608f676b7f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:04.437Z [INFO] server-3e681a2d-3c48-26ba-82ba-c3608f676b7f: entering candidate state: node="Node at 3e681a2d-3c48-26ba-82ba-c3608f676b7f [Candidate]" term=2 -2024-07-05T13:31:04.437Z [INFO] server-3e681a2d-3c48-26ba-82ba-c3608f676b7f: election won: term=2 tally=2 -2024-07-05T13:31:04.437Z [INFO] server-3e681a2d-3c48-26ba-82ba-c3608f676b7f: entering leader state: leader="Node at 3e681a2d-3c48-26ba-82ba-c3608f676b7f [Leader]" -2024-07-05T13:31:04.438Z [INFO] server-3e681a2d-3c48-26ba-82ba-c3608f676b7f: added peer, starting replication: peer=server-fbd8c553-0946-8822-1c7d-f33a6a289ead -2024-07-05T13:31:04.438Z [INFO] server-3e681a2d-3c48-26ba-82ba-c3608f676b7f: added peer, starting replication: peer=server-007edb95-fb76-8331-adce-852584231e8c -2024-07-05T13:31:04.438Z [INFO] server-3e681a2d-3c48-26ba-82ba-c3608f676b7f: pipelining replication: peer="{Voter server-007edb95-fb76-8331-adce-852584231e8c 007edb95-fb76-8331-adce-852584231e8c}" -2024-07-05T13:31:04.438Z [WARN] server-007edb95-fb76-8331-adce-852584231e8c: rejecting vote request since our last term is greater: candidate=3e681a2d-3c48-26ba-82ba-c3608f676b7f last-term=2 last-candidate-term=1 -2024-07-05T13:31:04.438Z [INFO] server-3e681a2d-3c48-26ba-82ba-c3608f676b7f: pipelining replication: peer="{Voter server-fbd8c553-0946-8822-1c7d-f33a6a289ead fbd8c553-0946-8822-1c7d-f33a6a289ead}" -2024-07-05T13:31:04.543Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:04.438394681 +0000 UTC m=+105.208305356 (1 nodes), 67.27976ms from start of poll, 67.972055ms from cluster start. Timeout at 2024-07-05 13:31:04.543434236 +0000 UTC m=+105.313344911, 105.039555ms after stability -2024-07-05T13:31:04.543Z [INFO] server-3e681a2d-3c48-26ba-82ba-c3608f676b7f: cannot transfer leadership to itself -2024-07-05T13:31:04.543Z [INFO] server-3e681a2d-3c48-26ba-82ba-c3608f676b7f: aborting pipeline replication: peer="{Voter server-007edb95-fb76-8331-adce-852584231e8c 007edb95-fb76-8331-adce-852584231e8c}" -2024-07-05T13:31:04.543Z [INFO] server-3e681a2d-3c48-26ba-82ba-c3608f676b7f: aborting pipeline replication: peer="{Voter server-fbd8c553-0946-8822-1c7d-f33a6a289ead fbd8c553-0946-8822-1c7d-f33a6a289ead}" -2024-07-05T13:31:05.044Z [ERROR] server-3e681a2d-3c48-26ba-82ba-c3608f676b7f: failed to heartbeat to: peer=fbd8c553-0946-8822-1c7d-f33a6a289ead backoff time=10ms error="command timed out" ---- PASS: TestRaft_LeadershipTransferToItself (0.68s) +2024-07-05T13:37:21.736Z [INFO] server-4d4968f1-468c-0e28-78ee-91d065136394: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d4968f1-468c-0e28-78ee-91d065136394 Address:4d4968f1-468c-0e28-78ee-91d065136394} {Suffrage:Voter ID:server-97218024-8325-126f-1b11-0c0032b88f6c Address:97218024-8325-126f-1b11-0c0032b88f6c} {Suffrage:Voter ID:server-fda7835d-2919-e35a-079e-793c59092a1e Address:fda7835d-2919-e35a-079e-793c59092a1e}]" +2024-07-05T13:37:21.737Z [INFO] server-4d4968f1-468c-0e28-78ee-91d065136394: entering follower state: follower="Node at 4d4968f1-468c-0e28-78ee-91d065136394 [Follower]" leader-address= leader-id= +2024-07-05T13:37:21.737Z [INFO] server-97218024-8325-126f-1b11-0c0032b88f6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d4968f1-468c-0e28-78ee-91d065136394 Address:4d4968f1-468c-0e28-78ee-91d065136394} {Suffrage:Voter ID:server-97218024-8325-126f-1b11-0c0032b88f6c Address:97218024-8325-126f-1b11-0c0032b88f6c} {Suffrage:Voter ID:server-fda7835d-2919-e35a-079e-793c59092a1e Address:fda7835d-2919-e35a-079e-793c59092a1e}]" +2024-07-05T13:37:21.737Z [INFO] server-97218024-8325-126f-1b11-0c0032b88f6c: entering follower state: follower="Node at 97218024-8325-126f-1b11-0c0032b88f6c [Follower]" leader-address= leader-id= +2024-07-05T13:37:21.737Z [INFO] server-fda7835d-2919-e35a-079e-793c59092a1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d4968f1-468c-0e28-78ee-91d065136394 Address:4d4968f1-468c-0e28-78ee-91d065136394} {Suffrage:Voter ID:server-97218024-8325-126f-1b11-0c0032b88f6c Address:97218024-8325-126f-1b11-0c0032b88f6c} {Suffrage:Voter ID:server-fda7835d-2919-e35a-079e-793c59092a1e Address:fda7835d-2919-e35a-079e-793c59092a1e}]" +2024-07-05T13:37:21.738Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:21.738Z [INFO] server-fda7835d-2919-e35a-079e-793c59092a1e: entering follower state: follower="Node at fda7835d-2919-e35a-079e-793c59092a1e [Follower]" leader-address= leader-id= +2024-07-05T13:37:21.788Z [WARN] server-97218024-8325-126f-1b11-0c0032b88f6c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:21.788Z [INFO] server-97218024-8325-126f-1b11-0c0032b88f6c: entering candidate state: node="Node at 97218024-8325-126f-1b11-0c0032b88f6c [Candidate]" term=2 +2024-07-05T13:37:21.789Z [INFO] server-97218024-8325-126f-1b11-0c0032b88f6c: election won: term=2 tally=2 +2024-07-05T13:37:21.789Z [INFO] server-97218024-8325-126f-1b11-0c0032b88f6c: entering leader state: leader="Node at 97218024-8325-126f-1b11-0c0032b88f6c [Leader]" +2024-07-05T13:37:21.789Z [INFO] server-97218024-8325-126f-1b11-0c0032b88f6c: added peer, starting replication: peer=server-4d4968f1-468c-0e28-78ee-91d065136394 +2024-07-05T13:37:21.789Z [INFO] server-97218024-8325-126f-1b11-0c0032b88f6c: added peer, starting replication: peer=server-fda7835d-2919-e35a-079e-793c59092a1e +2024-07-05T13:37:21.791Z [INFO] server-97218024-8325-126f-1b11-0c0032b88f6c: pipelining replication: peer="{Voter server-fda7835d-2919-e35a-079e-793c59092a1e fda7835d-2919-e35a-079e-793c59092a1e}" +2024-07-05T13:37:21.791Z [INFO] server-97218024-8325-126f-1b11-0c0032b88f6c: pipelining replication: peer="{Voter server-4d4968f1-468c-0e28-78ee-91d065136394 4d4968f1-468c-0e28-78ee-91d065136394}" +2024-07-05T13:37:21.896Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:21.790771685 +0000 UTC m=+107.786692845 (1 nodes), 52.678705ms from start of poll, 54.270593ms from cluster start. Timeout at 2024-07-05 13:37:21.89614076 +0000 UTC m=+107.892061920, 105.369075ms after stability +2024-07-05T13:37:21.896Z [INFO] server-97218024-8325-126f-1b11-0c0032b88f6c: cannot transfer leadership to itself +2024-07-05T13:37:21.896Z [INFO] server-97218024-8325-126f-1b11-0c0032b88f6c: aborting pipeline replication: peer="{Voter server-fda7835d-2919-e35a-079e-793c59092a1e fda7835d-2919-e35a-079e-793c59092a1e}" +2024-07-05T13:37:21.896Z [INFO] server-97218024-8325-126f-1b11-0c0032b88f6c: aborting pipeline replication: peer="{Voter server-4d4968f1-468c-0e28-78ee-91d065136394 4d4968f1-468c-0e28-78ee-91d065136394}" +--- PASS: TestRaft_LeadershipTransferToItself (0.17s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests -2024-07-05T13:31:05.046Z [INFO] server-c89bd3d2-cf3f-8ccb-8e62-001831c012c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c89bd3d2-cf3f-8ccb-8e62-001831c012c0 Address:c89bd3d2-cf3f-8ccb-8e62-001831c012c0} {Suffrage:Voter ID:server-e67c736d-be9f-3fd0-a13f-5192d2c5e0ae Address:e67c736d-be9f-3fd0-a13f-5192d2c5e0ae} {Suffrage:Voter ID:server-2c8ac418-846b-94b4-4a79-2d95359bbdcb Address:2c8ac418-846b-94b4-4a79-2d95359bbdcb}]" -2024-07-05T13:31:05.046Z [INFO] server-e67c736d-be9f-3fd0-a13f-5192d2c5e0ae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c89bd3d2-cf3f-8ccb-8e62-001831c012c0 Address:c89bd3d2-cf3f-8ccb-8e62-001831c012c0} {Suffrage:Voter ID:server-e67c736d-be9f-3fd0-a13f-5192d2c5e0ae Address:e67c736d-be9f-3fd0-a13f-5192d2c5e0ae} {Suffrage:Voter ID:server-2c8ac418-846b-94b4-4a79-2d95359bbdcb Address:2c8ac418-846b-94b4-4a79-2d95359bbdcb}]" -2024-07-05T13:31:05.046Z [INFO] server-2c8ac418-846b-94b4-4a79-2d95359bbdcb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c89bd3d2-cf3f-8ccb-8e62-001831c012c0 Address:c89bd3d2-cf3f-8ccb-8e62-001831c012c0} {Suffrage:Voter ID:server-e67c736d-be9f-3fd0-a13f-5192d2c5e0ae Address:e67c736d-be9f-3fd0-a13f-5192d2c5e0ae} {Suffrage:Voter ID:server-2c8ac418-846b-94b4-4a79-2d95359bbdcb Address:2c8ac418-846b-94b4-4a79-2d95359bbdcb}]" -2024-07-05T13:31:05.046Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:05.047Z [INFO] server-c89bd3d2-cf3f-8ccb-8e62-001831c012c0: entering follower state: follower="Node at c89bd3d2-cf3f-8ccb-8e62-001831c012c0 [Follower]" leader-address= leader-id= -2024-07-05T13:31:05.047Z [INFO] server-e67c736d-be9f-3fd0-a13f-5192d2c5e0ae: entering follower state: follower="Node at e67c736d-be9f-3fd0-a13f-5192d2c5e0ae [Follower]" leader-address= leader-id= -2024-07-05T13:31:05.047Z [INFO] server-2c8ac418-846b-94b4-4a79-2d95359bbdcb: entering follower state: follower="Node at 2c8ac418-846b-94b4-4a79-2d95359bbdcb [Follower]" leader-address= leader-id= -2024-07-05T13:31:05.112Z [WARN] server-2c8ac418-846b-94b4-4a79-2d95359bbdcb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:05.112Z [INFO] server-2c8ac418-846b-94b4-4a79-2d95359bbdcb: entering candidate state: node="Node at 2c8ac418-846b-94b4-4a79-2d95359bbdcb [Candidate]" term=2 -2024-07-05T13:31:05.113Z [INFO] server-2c8ac418-846b-94b4-4a79-2d95359bbdcb: election won: term=2 tally=2 -2024-07-05T13:31:05.113Z [INFO] server-2c8ac418-846b-94b4-4a79-2d95359bbdcb: entering leader state: leader="Node at 2c8ac418-846b-94b4-4a79-2d95359bbdcb [Leader]" -2024-07-05T13:31:05.113Z [INFO] server-2c8ac418-846b-94b4-4a79-2d95359bbdcb: added peer, starting replication: peer=server-c89bd3d2-cf3f-8ccb-8e62-001831c012c0 -2024-07-05T13:31:05.113Z [INFO] server-2c8ac418-846b-94b4-4a79-2d95359bbdcb: added peer, starting replication: peer=server-e67c736d-be9f-3fd0-a13f-5192d2c5e0ae -2024-07-05T13:31:05.113Z [WARN] server-e67c736d-be9f-3fd0-a13f-5192d2c5e0ae: rejecting vote request since our last term is greater: candidate=2c8ac418-846b-94b4-4a79-2d95359bbdcb last-term=2 last-candidate-term=1 -2024-07-05T13:31:05.113Z [INFO] server-2c8ac418-846b-94b4-4a79-2d95359bbdcb: pipelining replication: peer="{Voter server-c89bd3d2-cf3f-8ccb-8e62-001831c012c0 c89bd3d2-cf3f-8ccb-8e62-001831c012c0}" -2024-07-05T13:31:05.113Z [INFO] server-2c8ac418-846b-94b4-4a79-2d95359bbdcb: pipelining replication: peer="{Voter server-e67c736d-be9f-3fd0-a13f-5192d2c5e0ae e67c736d-be9f-3fd0-a13f-5192d2c5e0ae}" -2024-07-05T13:31:05.218Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:05.113547993 +0000 UTC m=+105.883458668 (1 nodes), 66.564624ms from start of poll, 67.197482ms from cluster start. Timeout at 2024-07-05 13:31:05.218635266 +0000 UTC m=+105.988545941, 105.087273ms after stability +2024-07-05T13:37:21.908Z [INFO] server-9ec934e8-5941-7142-48de-41d3137615d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ec934e8-5941-7142-48de-41d3137615d7 Address:9ec934e8-5941-7142-48de-41d3137615d7} {Suffrage:Voter ID:server-facf683d-be0b-a3d3-c271-0175774af9c6 Address:facf683d-be0b-a3d3-c271-0175774af9c6} {Suffrage:Voter ID:server-dfb19573-f115-3f5d-2664-e4fccae4f6fe Address:dfb19573-f115-3f5d-2664-e4fccae4f6fe}]" +2024-07-05T13:37:21.909Z [INFO] server-facf683d-be0b-a3d3-c271-0175774af9c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ec934e8-5941-7142-48de-41d3137615d7 Address:9ec934e8-5941-7142-48de-41d3137615d7} {Suffrage:Voter ID:server-facf683d-be0b-a3d3-c271-0175774af9c6 Address:facf683d-be0b-a3d3-c271-0175774af9c6} {Suffrage:Voter ID:server-dfb19573-f115-3f5d-2664-e4fccae4f6fe Address:dfb19573-f115-3f5d-2664-e4fccae4f6fe}]" +2024-07-05T13:37:21.909Z [INFO] server-dfb19573-f115-3f5d-2664-e4fccae4f6fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ec934e8-5941-7142-48de-41d3137615d7 Address:9ec934e8-5941-7142-48de-41d3137615d7} {Suffrage:Voter ID:server-facf683d-be0b-a3d3-c271-0175774af9c6 Address:facf683d-be0b-a3d3-c271-0175774af9c6} {Suffrage:Voter ID:server-dfb19573-f115-3f5d-2664-e4fccae4f6fe Address:dfb19573-f115-3f5d-2664-e4fccae4f6fe}]" +2024-07-05T13:37:21.909Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:21.910Z [INFO] server-9ec934e8-5941-7142-48de-41d3137615d7: entering follower state: follower="Node at 9ec934e8-5941-7142-48de-41d3137615d7 [Follower]" leader-address= leader-id= +2024-07-05T13:37:21.910Z [INFO] server-facf683d-be0b-a3d3-c271-0175774af9c6: entering follower state: follower="Node at facf683d-be0b-a3d3-c271-0175774af9c6 [Follower]" leader-address= leader-id= +2024-07-05T13:37:21.910Z [INFO] server-dfb19573-f115-3f5d-2664-e4fccae4f6fe: entering follower state: follower="Node at dfb19573-f115-3f5d-2664-e4fccae4f6fe [Follower]" leader-address= leader-id= +2024-07-05T13:37:21.969Z [WARN] server-facf683d-be0b-a3d3-c271-0175774af9c6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:21.969Z [INFO] server-facf683d-be0b-a3d3-c271-0175774af9c6: entering candidate state: node="Node at facf683d-be0b-a3d3-c271-0175774af9c6 [Candidate]" term=2 +2024-07-05T13:37:21.970Z [INFO] server-facf683d-be0b-a3d3-c271-0175774af9c6: election won: term=2 tally=2 +2024-07-05T13:37:21.970Z [INFO] server-facf683d-be0b-a3d3-c271-0175774af9c6: entering leader state: leader="Node at facf683d-be0b-a3d3-c271-0175774af9c6 [Leader]" +2024-07-05T13:37:21.970Z [INFO] server-facf683d-be0b-a3d3-c271-0175774af9c6: added peer, starting replication: peer=server-9ec934e8-5941-7142-48de-41d3137615d7 +2024-07-05T13:37:21.970Z [INFO] server-facf683d-be0b-a3d3-c271-0175774af9c6: added peer, starting replication: peer=server-dfb19573-f115-3f5d-2664-e4fccae4f6fe +2024-07-05T13:37:21.971Z [INFO] server-facf683d-be0b-a3d3-c271-0175774af9c6: pipelining replication: peer="{Voter server-9ec934e8-5941-7142-48de-41d3137615d7 9ec934e8-5941-7142-48de-41d3137615d7}" +2024-07-05T13:37:21.971Z [INFO] server-facf683d-be0b-a3d3-c271-0175774af9c6: pipelining replication: peer="{Voter server-dfb19573-f115-3f5d-2664-e4fccae4f6fe dfb19573-f115-3f5d-2664-e4fccae4f6fe}" +2024-07-05T13:37:22.075Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:21.970519944 +0000 UTC m=+107.966441104 (1 nodes), 60.573102ms from start of poll, 63.042599ms from cluster start. Timeout at 2024-07-05 13:37:22.075595317 +0000 UTC m=+108.071516477, 105.075373ms after stability raft_test.go:2457: waiting on future 0 raft_test.go:2457: waiting on future 1 raft_test.go:2457: waiting on future 2 @@ -3806,332 +3865,339 @@ raft_test.go:2457: waiting on future 4 raft_test.go:2457: waiting on future 5 raft_test.go:2457: waiting on future 6 -2024-07-05T13:31:05.219Z [INFO] server-2c8ac418-846b-94b4-4a79-2d95359bbdcb: aborting pipeline replication: peer="{Voter server-e67c736d-be9f-3fd0-a13f-5192d2c5e0ae e67c736d-be9f-3fd0-a13f-5192d2c5e0ae}" -2024-07-05T13:31:05.219Z [INFO] server-2c8ac418-846b-94b4-4a79-2d95359bbdcb: aborting pipeline replication: peer="{Voter server-c89bd3d2-cf3f-8ccb-8e62-001831c012c0 c89bd3d2-cf3f-8ccb-8e62-001831c012c0}" ---- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.17s) +2024-07-05T13:37:22.077Z [INFO] server-facf683d-be0b-a3d3-c271-0175774af9c6: aborting pipeline replication: peer="{Voter server-dfb19573-f115-3f5d-2664-e4fccae4f6fe dfb19573-f115-3f5d-2664-e4fccae4f6fe}" +2024-07-05T13:37:22.077Z [INFO] server-facf683d-be0b-a3d3-c271-0175774af9c6: aborting pipeline replication: peer="{Voter server-9ec934e8-5941-7142-48de-41d3137615d7 9ec934e8-5941-7142-48de-41d3137615d7}" +--- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.18s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout -2024-07-05T13:31:05.221Z [INFO] server-e242f040-f06a-7547-8424-9e8215771b16: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e242f040-f06a-7547-8424-9e8215771b16 Address:e242f040-f06a-7547-8424-9e8215771b16} {Suffrage:Voter ID:server-59a32e70-7438-5109-9e8d-4e6472c99aa5 Address:59a32e70-7438-5109-9e8d-4e6472c99aa5} {Suffrage:Voter ID:server-52ed6488-3b36-ff6d-b34e-c1770fe298c3 Address:52ed6488-3b36-ff6d-b34e-c1770fe298c3}]" -2024-07-05T13:31:05.221Z [INFO] server-59a32e70-7438-5109-9e8d-4e6472c99aa5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e242f040-f06a-7547-8424-9e8215771b16 Address:e242f040-f06a-7547-8424-9e8215771b16} {Suffrage:Voter ID:server-59a32e70-7438-5109-9e8d-4e6472c99aa5 Address:59a32e70-7438-5109-9e8d-4e6472c99aa5} {Suffrage:Voter ID:server-52ed6488-3b36-ff6d-b34e-c1770fe298c3 Address:52ed6488-3b36-ff6d-b34e-c1770fe298c3}]" -2024-07-05T13:31:05.222Z [INFO] server-52ed6488-3b36-ff6d-b34e-c1770fe298c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e242f040-f06a-7547-8424-9e8215771b16 Address:e242f040-f06a-7547-8424-9e8215771b16} {Suffrage:Voter ID:server-59a32e70-7438-5109-9e8d-4e6472c99aa5 Address:59a32e70-7438-5109-9e8d-4e6472c99aa5} {Suffrage:Voter ID:server-52ed6488-3b36-ff6d-b34e-c1770fe298c3 Address:52ed6488-3b36-ff6d-b34e-c1770fe298c3}]" -2024-07-05T13:31:05.222Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:05.222Z [INFO] server-e242f040-f06a-7547-8424-9e8215771b16: entering follower state: follower="Node at e242f040-f06a-7547-8424-9e8215771b16 [Follower]" leader-address= leader-id= -2024-07-05T13:31:05.222Z [INFO] server-59a32e70-7438-5109-9e8d-4e6472c99aa5: entering follower state: follower="Node at 59a32e70-7438-5109-9e8d-4e6472c99aa5 [Follower]" leader-address= leader-id= -2024-07-05T13:31:05.222Z [INFO] server-52ed6488-3b36-ff6d-b34e-c1770fe298c3: entering follower state: follower="Node at 52ed6488-3b36-ff6d-b34e-c1770fe298c3 [Follower]" leader-address= leader-id= -2024-07-05T13:31:05.280Z [WARN] server-52ed6488-3b36-ff6d-b34e-c1770fe298c3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:05.280Z [INFO] server-52ed6488-3b36-ff6d-b34e-c1770fe298c3: entering candidate state: node="Node at 52ed6488-3b36-ff6d-b34e-c1770fe298c3 [Candidate]" term=2 -2024-07-05T13:31:05.280Z [INFO] server-52ed6488-3b36-ff6d-b34e-c1770fe298c3: election won: term=2 tally=2 -2024-07-05T13:31:05.280Z [INFO] server-52ed6488-3b36-ff6d-b34e-c1770fe298c3: entering leader state: leader="Node at 52ed6488-3b36-ff6d-b34e-c1770fe298c3 [Leader]" -2024-07-05T13:31:05.280Z [INFO] server-52ed6488-3b36-ff6d-b34e-c1770fe298c3: added peer, starting replication: peer=server-e242f040-f06a-7547-8424-9e8215771b16 -2024-07-05T13:31:05.280Z [INFO] server-52ed6488-3b36-ff6d-b34e-c1770fe298c3: added peer, starting replication: peer=server-59a32e70-7438-5109-9e8d-4e6472c99aa5 -2024-07-05T13:31:05.281Z [INFO] server-52ed6488-3b36-ff6d-b34e-c1770fe298c3: pipelining replication: peer="{Voter server-59a32e70-7438-5109-9e8d-4e6472c99aa5 59a32e70-7438-5109-9e8d-4e6472c99aa5}" -2024-07-05T13:31:05.281Z [WARN] server-59a32e70-7438-5109-9e8d-4e6472c99aa5: rejecting vote request since our last term is greater: candidate=52ed6488-3b36-ff6d-b34e-c1770fe298c3 last-term=2 last-candidate-term=1 -2024-07-05T13:31:05.281Z [INFO] server-52ed6488-3b36-ff6d-b34e-c1770fe298c3: pipelining replication: peer="{Voter server-e242f040-f06a-7547-8424-9e8215771b16 e242f040-f06a-7547-8424-9e8215771b16}" -2024-07-05T13:31:05.393Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:05.281250549 +0000 UTC m=+106.051161224 (1 nodes), 59.072487ms from start of poll, 59.817341ms from cluster start. Timeout at 2024-07-05 13:31:05.387167713 +0000 UTC m=+106.157078388, 105.917164ms after stability -2024-07-05T13:31:05.393Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:31:05.505Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:31:05.393987794 +0000 UTC m=+106.163898469 (2 nodes), 1.92µs from start of poll, 172.554586ms from cluster start. Timeout at 2024-07-05 13:31:05.505372286 +0000 UTC m=+106.275282961, 111.384492ms after stability -2024-07-05T13:31:07.170Z [INFO] server-52ed6488-3b36-ff6d-b34e-c1770fe298c3: aborting pipeline replication: peer="{Voter server-59a32e70-7438-5109-9e8d-4e6472c99aa5 59a32e70-7438-5109-9e8d-4e6472c99aa5}" -2024-07-05T13:31:07.170Z [INFO] server-52ed6488-3b36-ff6d-b34e-c1770fe298c3: aborting pipeline replication: peer="{Voter server-e242f040-f06a-7547-8424-9e8215771b16 e242f040-f06a-7547-8424-9e8215771b16}" ---- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (1.95s) +2024-07-05T13:37:22.087Z [INFO] server-f144eaca-b1e8-d5d8-5ed3-06365447bd0a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f144eaca-b1e8-d5d8-5ed3-06365447bd0a Address:f144eaca-b1e8-d5d8-5ed3-06365447bd0a} {Suffrage:Voter ID:server-9af18852-eec7-51b2-3525-2edf2d0c4c42 Address:9af18852-eec7-51b2-3525-2edf2d0c4c42} {Suffrage:Voter ID:server-48851bbb-13c4-b9f9-96ea-63cb70dee0b3 Address:48851bbb-13c4-b9f9-96ea-63cb70dee0b3}]" +2024-07-05T13:37:22.087Z [INFO] server-f144eaca-b1e8-d5d8-5ed3-06365447bd0a: entering follower state: follower="Node at f144eaca-b1e8-d5d8-5ed3-06365447bd0a [Follower]" leader-address= leader-id= +2024-07-05T13:37:22.088Z [INFO] server-9af18852-eec7-51b2-3525-2edf2d0c4c42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f144eaca-b1e8-d5d8-5ed3-06365447bd0a Address:f144eaca-b1e8-d5d8-5ed3-06365447bd0a} {Suffrage:Voter ID:server-9af18852-eec7-51b2-3525-2edf2d0c4c42 Address:9af18852-eec7-51b2-3525-2edf2d0c4c42} {Suffrage:Voter ID:server-48851bbb-13c4-b9f9-96ea-63cb70dee0b3 Address:48851bbb-13c4-b9f9-96ea-63cb70dee0b3}]" +2024-07-05T13:37:22.088Z [INFO] server-9af18852-eec7-51b2-3525-2edf2d0c4c42: entering follower state: follower="Node at 9af18852-eec7-51b2-3525-2edf2d0c4c42 [Follower]" leader-address= leader-id= +2024-07-05T13:37:22.089Z [INFO] server-48851bbb-13c4-b9f9-96ea-63cb70dee0b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f144eaca-b1e8-d5d8-5ed3-06365447bd0a Address:f144eaca-b1e8-d5d8-5ed3-06365447bd0a} {Suffrage:Voter ID:server-9af18852-eec7-51b2-3525-2edf2d0c4c42 Address:9af18852-eec7-51b2-3525-2edf2d0c4c42} {Suffrage:Voter ID:server-48851bbb-13c4-b9f9-96ea-63cb70dee0b3 Address:48851bbb-13c4-b9f9-96ea-63cb70dee0b3}]" +2024-07-05T13:37:22.089Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:22.089Z [INFO] server-48851bbb-13c4-b9f9-96ea-63cb70dee0b3: entering follower state: follower="Node at 48851bbb-13c4-b9f9-96ea-63cb70dee0b3 [Follower]" leader-address= leader-id= +2024-07-05T13:37:22.158Z [WARN] server-f144eaca-b1e8-d5d8-5ed3-06365447bd0a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:22.159Z [INFO] server-f144eaca-b1e8-d5d8-5ed3-06365447bd0a: entering candidate state: node="Node at f144eaca-b1e8-d5d8-5ed3-06365447bd0a [Candidate]" term=2 +2024-07-05T13:37:22.159Z [WARN] server-9af18852-eec7-51b2-3525-2edf2d0c4c42: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:22.159Z [INFO] server-9af18852-eec7-51b2-3525-2edf2d0c4c42: entering candidate state: node="Node at 9af18852-eec7-51b2-3525-2edf2d0c4c42 [Candidate]" term=2 +2024-07-05T13:37:22.160Z [INFO] server-f144eaca-b1e8-d5d8-5ed3-06365447bd0a: duplicate requestVote for same term: term=2 +2024-07-05T13:37:22.160Z [INFO] server-48851bbb-13c4-b9f9-96ea-63cb70dee0b3: duplicate requestVote for same term: term=2 +2024-07-05T13:37:22.160Z [INFO] server-9af18852-eec7-51b2-3525-2edf2d0c4c42: duplicate requestVote for same term: term=2 +2024-07-05T13:37:22.160Z [INFO] server-9af18852-eec7-51b2-3525-2edf2d0c4c42: election won: term=2 tally=2 +2024-07-05T13:37:22.160Z [INFO] server-9af18852-eec7-51b2-3525-2edf2d0c4c42: entering leader state: leader="Node at 9af18852-eec7-51b2-3525-2edf2d0c4c42 [Leader]" +2024-07-05T13:37:22.160Z [INFO] server-9af18852-eec7-51b2-3525-2edf2d0c4c42: added peer, starting replication: peer=server-f144eaca-b1e8-d5d8-5ed3-06365447bd0a +2024-07-05T13:37:22.160Z [INFO] server-9af18852-eec7-51b2-3525-2edf2d0c4c42: added peer, starting replication: peer=server-48851bbb-13c4-b9f9-96ea-63cb70dee0b3 +2024-07-05T13:37:22.161Z [INFO] server-f144eaca-b1e8-d5d8-5ed3-06365447bd0a: entering follower state: follower="Node at f144eaca-b1e8-d5d8-5ed3-06365447bd0a [Follower]" leader-address=9af18852-eec7-51b2-3525-2edf2d0c4c42 leader-id=server-9af18852-eec7-51b2-3525-2edf2d0c4c42 +2024-07-05T13:37:22.161Z [INFO] server-9af18852-eec7-51b2-3525-2edf2d0c4c42: pipelining replication: peer="{Voter server-48851bbb-13c4-b9f9-96ea-63cb70dee0b3 48851bbb-13c4-b9f9-96ea-63cb70dee0b3}" +2024-07-05T13:37:22.161Z [INFO] server-9af18852-eec7-51b2-3525-2edf2d0c4c42: pipelining replication: peer="{Voter server-f144eaca-b1e8-d5d8-5ed3-06365447bd0a f144eaca-b1e8-d5d8-5ed3-06365447bd0a}" +2024-07-05T13:37:22.267Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:22.161594871 +0000 UTC m=+108.157516031 (1 nodes), 72.061934ms from start of poll, 75.08967ms from cluster start. Timeout at 2024-07-05 13:37:22.266984072 +0000 UTC m=+108.262904940, 105.388909ms after stability +2024-07-05T13:37:22.267Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:22.372Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:22.267704184 +0000 UTC m=+108.263625344 (2 nodes), 5.833µs from start of poll, 181.198983ms from cluster start. Timeout at 2024-07-05 13:37:22.372720933 +0000 UTC m=+108.368642093, 105.016749ms after stability +2024-07-05T13:37:23.981Z [INFO] server-9af18852-eec7-51b2-3525-2edf2d0c4c42: aborting pipeline replication: peer="{Voter server-48851bbb-13c4-b9f9-96ea-63cb70dee0b3 48851bbb-13c4-b9f9-96ea-63cb70dee0b3}" +2024-07-05T13:37:23.981Z [INFO] server-9af18852-eec7-51b2-3525-2edf2d0c4c42: aborting pipeline replication: peer="{Voter server-f144eaca-b1e8-d5d8-5ed3-06365447bd0a f144eaca-b1e8-d5d8-5ed3-06365447bd0a}" +--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (1.90s) === RUN TestRaft_LeadershipTransferIgnoresNonvoters -2024-07-05T13:31:07.172Z [INFO] server-c0528ce2-073c-053e-0da9-1ec47aa564c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0528ce2-073c-053e-0da9-1ec47aa564c2 Address:c0528ce2-073c-053e-0da9-1ec47aa564c2} {Suffrage:Voter ID:server-c704c264-8c6e-83e0-8fde-9c9b6b576edc Address:c704c264-8c6e-83e0-8fde-9c9b6b576edc}]" -2024-07-05T13:31:07.173Z [INFO] server-c704c264-8c6e-83e0-8fde-9c9b6b576edc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0528ce2-073c-053e-0da9-1ec47aa564c2 Address:c0528ce2-073c-053e-0da9-1ec47aa564c2} {Suffrage:Voter ID:server-c704c264-8c6e-83e0-8fde-9c9b6b576edc Address:c704c264-8c6e-83e0-8fde-9c9b6b576edc}]" -2024-07-05T13:31:07.173Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:31:07.173Z [INFO] server-c0528ce2-073c-053e-0da9-1ec47aa564c2: entering follower state: follower="Node at c0528ce2-073c-053e-0da9-1ec47aa564c2 [Follower]" leader-address= leader-id= -2024-07-05T13:31:07.173Z [INFO] server-c704c264-8c6e-83e0-8fde-9c9b6b576edc: entering follower state: follower="Node at c704c264-8c6e-83e0-8fde-9c9b6b576edc [Follower]" leader-address= leader-id= -2024-07-05T13:31:07.264Z [WARN] server-c704c264-8c6e-83e0-8fde-9c9b6b576edc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:07.264Z [INFO] server-c704c264-8c6e-83e0-8fde-9c9b6b576edc: entering candidate state: node="Node at c704c264-8c6e-83e0-8fde-9c9b6b576edc [Candidate]" term=2 -2024-07-05T13:31:07.264Z [INFO] server-c704c264-8c6e-83e0-8fde-9c9b6b576edc: election won: term=2 tally=2 -2024-07-05T13:31:07.264Z [INFO] server-c704c264-8c6e-83e0-8fde-9c9b6b576edc: entering leader state: leader="Node at c704c264-8c6e-83e0-8fde-9c9b6b576edc [Leader]" -2024-07-05T13:31:07.264Z [INFO] server-c704c264-8c6e-83e0-8fde-9c9b6b576edc: added peer, starting replication: peer=server-c0528ce2-073c-053e-0da9-1ec47aa564c2 -2024-07-05T13:31:07.265Z [INFO] server-c704c264-8c6e-83e0-8fde-9c9b6b576edc: pipelining replication: peer="{Voter server-c0528ce2-073c-053e-0da9-1ec47aa564c2 c0528ce2-073c-053e-0da9-1ec47aa564c2}" -2024-07-05T13:31:07.370Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:31:07.26501111 +0000 UTC m=+108.034921785 (1 nodes), 91.656385ms from start of poll, 92.241764ms from cluster start. Timeout at 2024-07-05 13:31:07.370534207 +0000 UTC m=+108.140444882, 105.523097ms after stability -2024-07-05T13:31:07.370Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:07.477Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:07.370696482 +0000 UTC m=+108.140607157 (1 nodes), 1.24µs from start of poll, 197.927136ms from cluster start. Timeout at 2024-07-05 13:31:07.477028671 +0000 UTC m=+108.246939306, 106.332149ms after stability -2024-07-05T13:31:07.477Z [INFO] server-c704c264-8c6e-83e0-8fde-9c9b6b576edc: updating configuration: command=DemoteVoter server-id=server-c0528ce2-073c-053e-0da9-1ec47aa564c2 server-addr= servers="[{Suffrage:Nonvoter ID:server-c0528ce2-073c-053e-0da9-1ec47aa564c2 Address:c0528ce2-073c-053e-0da9-1ec47aa564c2} {Suffrage:Voter ID:server-c704c264-8c6e-83e0-8fde-9c9b6b576edc Address:c704c264-8c6e-83e0-8fde-9c9b6b576edc}]" -2024-07-05T13:31:07.477Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:07.595Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:07.477753406 +0000 UTC m=+108.247664121 (1 nodes), 1.28µs from start of poll, 304.9841ms from cluster start. Timeout at 2024-07-05 13:31:07.583199146 +0000 UTC m=+108.353109821, 105.4457ms after stability -2024-07-05T13:31:07.596Z [INFO] server-c704c264-8c6e-83e0-8fde-9c9b6b576edc: aborting pipeline replication: peer="{Voter server-c0528ce2-073c-053e-0da9-1ec47aa564c2 c0528ce2-073c-053e-0da9-1ec47aa564c2}" ---- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.43s) +2024-07-05T13:37:23.984Z [INFO] server-9e1769f8-7bf0-267d-81dd-cf15ed526d46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e1769f8-7bf0-267d-81dd-cf15ed526d46 Address:9e1769f8-7bf0-267d-81dd-cf15ed526d46} {Suffrage:Voter ID:server-ac88b0b8-19e2-4a1f-2806-76ee271a62fb Address:ac88b0b8-19e2-4a1f-2806-76ee271a62fb}]" +2024-07-05T13:37:23.984Z [INFO] server-ac88b0b8-19e2-4a1f-2806-76ee271a62fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e1769f8-7bf0-267d-81dd-cf15ed526d46 Address:9e1769f8-7bf0-267d-81dd-cf15ed526d46} {Suffrage:Voter ID:server-ac88b0b8-19e2-4a1f-2806-76ee271a62fb Address:ac88b0b8-19e2-4a1f-2806-76ee271a62fb}]" +2024-07-05T13:37:23.984Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:23.984Z [INFO] server-9e1769f8-7bf0-267d-81dd-cf15ed526d46: entering follower state: follower="Node at 9e1769f8-7bf0-267d-81dd-cf15ed526d46 [Follower]" leader-address= leader-id= +2024-07-05T13:37:23.984Z [INFO] server-ac88b0b8-19e2-4a1f-2806-76ee271a62fb: entering follower state: follower="Node at ac88b0b8-19e2-4a1f-2806-76ee271a62fb [Follower]" leader-address= leader-id= +2024-07-05T13:37:24.051Z [WARN] server-ac88b0b8-19e2-4a1f-2806-76ee271a62fb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:24.051Z [INFO] server-ac88b0b8-19e2-4a1f-2806-76ee271a62fb: entering candidate state: node="Node at ac88b0b8-19e2-4a1f-2806-76ee271a62fb [Candidate]" term=2 +2024-07-05T13:37:24.051Z [INFO] server-ac88b0b8-19e2-4a1f-2806-76ee271a62fb: election won: term=2 tally=2 +2024-07-05T13:37:24.051Z [INFO] server-ac88b0b8-19e2-4a1f-2806-76ee271a62fb: entering leader state: leader="Node at ac88b0b8-19e2-4a1f-2806-76ee271a62fb [Leader]" +2024-07-05T13:37:24.051Z [INFO] server-ac88b0b8-19e2-4a1f-2806-76ee271a62fb: added peer, starting replication: peer=server-9e1769f8-7bf0-267d-81dd-cf15ed526d46 +2024-07-05T13:37:24.052Z [INFO] server-ac88b0b8-19e2-4a1f-2806-76ee271a62fb: pipelining replication: peer="{Voter server-9e1769f8-7bf0-267d-81dd-cf15ed526d46 9e1769f8-7bf0-267d-81dd-cf15ed526d46}" +2024-07-05T13:37:24.157Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:24.051820522 +0000 UTC m=+110.047741099 (1 nodes), 66.990527ms from start of poll, 67.565683ms from cluster start. Timeout at 2024-07-05 13:37:24.157094516 +0000 UTC m=+110.153015676, 105.274577ms after stability +2024-07-05T13:37:24.157Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:24.262Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:24.157424677 +0000 UTC m=+110.153345545 (1 nodes), 3.5µs from start of poll, 173.170129ms from cluster start. Timeout at 2024-07-05 13:37:24.262675047 +0000 UTC m=+110.258595915, 105.25037ms after stability +2024-07-05T13:37:24.263Z [INFO] server-ac88b0b8-19e2-4a1f-2806-76ee271a62fb: updating configuration: command=DemoteVoter server-id=server-9e1769f8-7bf0-267d-81dd-cf15ed526d46 server-addr= servers="[{Suffrage:Nonvoter ID:server-9e1769f8-7bf0-267d-81dd-cf15ed526d46 Address:9e1769f8-7bf0-267d-81dd-cf15ed526d46} {Suffrage:Voter ID:server-ac88b0b8-19e2-4a1f-2806-76ee271a62fb Address:ac88b0b8-19e2-4a1f-2806-76ee271a62fb}]" +2024-07-05T13:37:24.264Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:24.369Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:24.264231643 +0000 UTC m=+110.260152511 (1 nodes), 5.249µs from start of poll, 279.977095ms from cluster start. Timeout at 2024-07-05 13:37:24.369330641 +0000 UTC m=+110.365252092, 105.099581ms after stability +2024-07-05T13:37:24.369Z [INFO] server-ac88b0b8-19e2-4a1f-2806-76ee271a62fb: aborting pipeline replication: peer="{Voter server-9e1769f8-7bf0-267d-81dd-cf15ed526d46 9e1769f8-7bf0-267d-81dd-cf15ed526d46}" +--- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.39s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap -2024-07-05T13:31:07.598Z [INFO] server-d82dcb0f-b688-4bdd-5219-2e638b4b8106: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d82dcb0f-b688-4bdd-5219-2e638b4b8106 Address:d82dcb0f-b688-4bdd-5219-2e638b4b8106} {Suffrage:Voter ID:server-e0e97859-dc53-5bec-1feb-c75831fc636e Address:e0e97859-dc53-5bec-1feb-c75831fc636e}]" -2024-07-05T13:31:07.598Z [INFO] server-e0e97859-dc53-5bec-1feb-c75831fc636e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d82dcb0f-b688-4bdd-5219-2e638b4b8106 Address:d82dcb0f-b688-4bdd-5219-2e638b4b8106} {Suffrage:Voter ID:server-e0e97859-dc53-5bec-1feb-c75831fc636e Address:e0e97859-dc53-5bec-1feb-c75831fc636e}]" -2024-07-05T13:31:07.598Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:31:07.598Z [INFO] server-d82dcb0f-b688-4bdd-5219-2e638b4b8106: entering follower state: follower="Node at d82dcb0f-b688-4bdd-5219-2e638b4b8106 [Follower]" leader-address= leader-id= -2024-07-05T13:31:07.598Z [INFO] server-e0e97859-dc53-5bec-1feb-c75831fc636e: entering follower state: follower="Node at e0e97859-dc53-5bec-1feb-c75831fc636e [Follower]" leader-address= leader-id= -2024-07-05T13:31:07.654Z [WARN] server-d82dcb0f-b688-4bdd-5219-2e638b4b8106: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:07.654Z [INFO] server-d82dcb0f-b688-4bdd-5219-2e638b4b8106: entering candidate state: node="Node at d82dcb0f-b688-4bdd-5219-2e638b4b8106 [Candidate]" term=2 -2024-07-05T13:31:07.654Z [INFO] server-d82dcb0f-b688-4bdd-5219-2e638b4b8106: election won: term=2 tally=2 -2024-07-05T13:31:07.654Z [INFO] server-d82dcb0f-b688-4bdd-5219-2e638b4b8106: entering leader state: leader="Node at d82dcb0f-b688-4bdd-5219-2e638b4b8106 [Leader]" -2024-07-05T13:31:07.655Z [INFO] server-d82dcb0f-b688-4bdd-5219-2e638b4b8106: added peer, starting replication: peer=server-e0e97859-dc53-5bec-1feb-c75831fc636e -2024-07-05T13:31:07.655Z [INFO] server-d82dcb0f-b688-4bdd-5219-2e638b4b8106: pipelining replication: peer="{Voter server-e0e97859-dc53-5bec-1feb-c75831fc636e e0e97859-dc53-5bec-1feb-c75831fc636e}" -2024-07-05T13:31:07.760Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:31:07.655326656 +0000 UTC m=+108.425237331 (1 nodes), 56.811047ms from start of poll, 57.260631ms from cluster start. Timeout at 2024-07-05 13:31:07.760339891 +0000 UTC m=+108.530250566, 105.013235ms after stability -2024-07-05T13:31:07.760Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:07.870Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:07.760565163 +0000 UTC m=+108.530475838 (1 nodes), 1.48µs from start of poll, 162.499138ms from cluster start. Timeout at 2024-07-05 13:31:07.870252355 +0000 UTC m=+108.640163030, 109.687192ms after stability -2024-07-05T13:31:07.881Z [INFO] server-e0e97859-dc53-5bec-1feb-c75831fc636e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d82dcb0f-b688-4bdd-5219-2e638b4b8106 Address:d82dcb0f-b688-4bdd-5219-2e638b4b8106} {Suffrage:Voter ID:server-e0e97859-dc53-5bec-1feb-c75831fc636e Address:e0e97859-dc53-5bec-1feb-c75831fc636e}]" -2024-07-05T13:31:07.881Z [INFO] server-d82dcb0f-b688-4bdd-5219-2e638b4b8106: aborting pipeline replication: peer="{Voter server-e0e97859-dc53-5bec-1feb-c75831fc636e e0e97859-dc53-5bec-1feb-c75831fc636e}" ---- PASS: TestRaft_GetConfigurationNoBootstrap (0.29s) +2024-07-05T13:37:24.376Z [INFO] server-37f8794f-5471-1c7b-54a1-a6d9da5da694: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37f8794f-5471-1c7b-54a1-a6d9da5da694 Address:37f8794f-5471-1c7b-54a1-a6d9da5da694} {Suffrage:Voter ID:server-d53dfa25-243e-af6f-76cd-07315097316b Address:d53dfa25-243e-af6f-76cd-07315097316b}]" +2024-07-05T13:37:24.376Z [INFO] server-37f8794f-5471-1c7b-54a1-a6d9da5da694: entering follower state: follower="Node at 37f8794f-5471-1c7b-54a1-a6d9da5da694 [Follower]" leader-address= leader-id= +2024-07-05T13:37:24.376Z [INFO] server-d53dfa25-243e-af6f-76cd-07315097316b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37f8794f-5471-1c7b-54a1-a6d9da5da694 Address:37f8794f-5471-1c7b-54a1-a6d9da5da694} {Suffrage:Voter ID:server-d53dfa25-243e-af6f-76cd-07315097316b Address:d53dfa25-243e-af6f-76cd-07315097316b}]" +2024-07-05T13:37:24.376Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:24.377Z [INFO] server-d53dfa25-243e-af6f-76cd-07315097316b: entering follower state: follower="Node at d53dfa25-243e-af6f-76cd-07315097316b [Follower]" leader-address= leader-id= +2024-07-05T13:37:24.437Z [WARN] server-d53dfa25-243e-af6f-76cd-07315097316b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:24.437Z [INFO] server-d53dfa25-243e-af6f-76cd-07315097316b: entering candidate state: node="Node at d53dfa25-243e-af6f-76cd-07315097316b [Candidate]" term=2 +2024-07-05T13:37:24.437Z [INFO] server-d53dfa25-243e-af6f-76cd-07315097316b: election won: term=2 tally=2 +2024-07-05T13:37:24.437Z [INFO] server-d53dfa25-243e-af6f-76cd-07315097316b: entering leader state: leader="Node at d53dfa25-243e-af6f-76cd-07315097316b [Leader]" +2024-07-05T13:37:24.437Z [INFO] server-d53dfa25-243e-af6f-76cd-07315097316b: added peer, starting replication: peer=server-37f8794f-5471-1c7b-54a1-a6d9da5da694 +2024-07-05T13:37:24.438Z [INFO] server-d53dfa25-243e-af6f-76cd-07315097316b: pipelining replication: peer="{Voter server-37f8794f-5471-1c7b-54a1-a6d9da5da694 37f8794f-5471-1c7b-54a1-a6d9da5da694}" +2024-07-05T13:37:24.543Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:24.437779474 +0000 UTC m=+110.433700342 (1 nodes), 60.810514ms from start of poll, 62.108115ms from cluster start. Timeout at 2024-07-05 13:37:24.543358255 +0000 UTC m=+110.539279123, 105.578781ms after stability +2024-07-05T13:37:24.543Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:24.648Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:24.543689582 +0000 UTC m=+110.539610158 (1 nodes), 2.916µs from start of poll, 168.017931ms from cluster start. Timeout at 2024-07-05 13:37:24.648783038 +0000 UTC m=+110.644704489, 105.094331ms after stability +2024-07-05T13:37:24.674Z [INFO] server-d53dfa25-243e-af6f-76cd-07315097316b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37f8794f-5471-1c7b-54a1-a6d9da5da694 Address:37f8794f-5471-1c7b-54a1-a6d9da5da694} {Suffrage:Voter ID:server-d53dfa25-243e-af6f-76cd-07315097316b Address:d53dfa25-243e-af6f-76cd-07315097316b}]" +2024-07-05T13:37:24.674Z [INFO] server-d53dfa25-243e-af6f-76cd-07315097316b: aborting pipeline replication: peer="{Voter server-37f8794f-5471-1c7b-54a1-a6d9da5da694 37f8794f-5471-1c7b-54a1-a6d9da5da694}" +--- PASS: TestRaft_GetConfigurationNoBootstrap (0.31s) === RUN TestRaft_LogStoreIsMonotonic -2024-07-05T13:31:07.883Z [INFO] server-7e4dfabd-53f9-3707-b1f3-e63b0985c92a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e4dfabd-53f9-3707-b1f3-e63b0985c92a Address:7e4dfabd-53f9-3707-b1f3-e63b0985c92a}]" -2024-07-05T13:31:07.883Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:07.883Z [INFO] server-7e4dfabd-53f9-3707-b1f3-e63b0985c92a: entering follower state: follower="Node at 7e4dfabd-53f9-3707-b1f3-e63b0985c92a [Follower]" leader-address= leader-id= -2024-07-05T13:31:07.960Z [WARN] server-7e4dfabd-53f9-3707-b1f3-e63b0985c92a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:07.960Z [INFO] server-7e4dfabd-53f9-3707-b1f3-e63b0985c92a: entering candidate state: node="Node at 7e4dfabd-53f9-3707-b1f3-e63b0985c92a [Candidate]" term=2 -2024-07-05T13:31:07.961Z [INFO] server-7e4dfabd-53f9-3707-b1f3-e63b0985c92a: election won: term=2 tally=1 -2024-07-05T13:31:07.961Z [INFO] server-7e4dfabd-53f9-3707-b1f3-e63b0985c92a: entering leader state: leader="Node at 7e4dfabd-53f9-3707-b1f3-e63b0985c92a [Leader]" -2024-07-05T13:31:08.066Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:07.961226043 +0000 UTC m=+108.731136718 (1 nodes), 77.738072ms from start of poll, 77.999663ms from cluster start. Timeout at 2024-07-05 13:31:08.066329396 +0000 UTC m=+108.836240071, 105.103353ms after stability ---- PASS: TestRaft_LogStoreIsMonotonic (0.18s) +2024-07-05T13:37:24.682Z [INFO] server-3e9e63e2-47ec-07be-0a0a-51ff54d25565: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e9e63e2-47ec-07be-0a0a-51ff54d25565 Address:3e9e63e2-47ec-07be-0a0a-51ff54d25565}]" +2024-07-05T13:37:24.682Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:24.682Z [INFO] server-3e9e63e2-47ec-07be-0a0a-51ff54d25565: entering follower state: follower="Node at 3e9e63e2-47ec-07be-0a0a-51ff54d25565 [Follower]" leader-address= leader-id= +2024-07-05T13:37:24.758Z [WARN] server-3e9e63e2-47ec-07be-0a0a-51ff54d25565: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:24.759Z [INFO] server-3e9e63e2-47ec-07be-0a0a-51ff54d25565: entering candidate state: node="Node at 3e9e63e2-47ec-07be-0a0a-51ff54d25565 [Candidate]" term=2 +2024-07-05T13:37:24.759Z [INFO] server-3e9e63e2-47ec-07be-0a0a-51ff54d25565: election won: term=2 tally=1 +2024-07-05T13:37:24.759Z [INFO] server-3e9e63e2-47ec-07be-0a0a-51ff54d25565: entering leader state: leader="Node at 3e9e63e2-47ec-07be-0a0a-51ff54d25565 [Leader]" +2024-07-05T13:37:24.865Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:24.759894342 +0000 UTC m=+110.755815794 (1 nodes), 77.345963ms from start of poll, 78.066075ms from cluster start. Timeout at 2024-07-05 13:37:24.864973507 +0000 UTC m=+110.860894375, 105.078581ms after stability +--- PASS: TestRaft_LogStoreIsMonotonic (0.19s) === RUN TestRaft_CacheLogWithStoreError -2024-07-05T13:31:08.067Z [INFO] server-50c17783-5ed4-7593-8f8f-97489bf931aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50c17783-5ed4-7593-8f8f-97489bf931aa Address:50c17783-5ed4-7593-8f8f-97489bf931aa} {Suffrage:Voter ID:server-45ab51d5-5a7f-bedd-6b4f-119ffe74236b Address:45ab51d5-5a7f-bedd-6b4f-119ffe74236b}]" -2024-07-05T13:31:08.068Z [INFO] server-45ab51d5-5a7f-bedd-6b4f-119ffe74236b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50c17783-5ed4-7593-8f8f-97489bf931aa Address:50c17783-5ed4-7593-8f8f-97489bf931aa} {Suffrage:Voter ID:server-45ab51d5-5a7f-bedd-6b4f-119ffe74236b Address:45ab51d5-5a7f-bedd-6b4f-119ffe74236b}]" -2024-07-05T13:31:08.068Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:31:08.068Z [INFO] server-50c17783-5ed4-7593-8f8f-97489bf931aa: entering follower state: follower="Node at 50c17783-5ed4-7593-8f8f-97489bf931aa [Follower]" leader-address= leader-id= -2024-07-05T13:31:08.068Z [INFO] server-45ab51d5-5a7f-bedd-6b4f-119ffe74236b: entering follower state: follower="Node at 45ab51d5-5a7f-bedd-6b4f-119ffe74236b [Follower]" leader-address= leader-id= -2024-07-05T13:31:08.145Z [WARN] server-45ab51d5-5a7f-bedd-6b4f-119ffe74236b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:08.145Z [INFO] server-45ab51d5-5a7f-bedd-6b4f-119ffe74236b: entering candidate state: node="Node at 45ab51d5-5a7f-bedd-6b4f-119ffe74236b [Candidate]" term=2 -2024-07-05T13:31:08.146Z [INFO] server-45ab51d5-5a7f-bedd-6b4f-119ffe74236b: election won: term=2 tally=2 -2024-07-05T13:31:08.146Z [INFO] server-45ab51d5-5a7f-bedd-6b4f-119ffe74236b: entering leader state: leader="Node at 45ab51d5-5a7f-bedd-6b4f-119ffe74236b [Leader]" -2024-07-05T13:31:08.146Z [INFO] server-45ab51d5-5a7f-bedd-6b4f-119ffe74236b: added peer, starting replication: peer=server-50c17783-5ed4-7593-8f8f-97489bf931aa -2024-07-05T13:31:08.146Z [INFO] server-45ab51d5-5a7f-bedd-6b4f-119ffe74236b: pipelining replication: peer="{Voter server-50c17783-5ed4-7593-8f8f-97489bf931aa 50c17783-5ed4-7593-8f8f-97489bf931aa}" -2024-07-05T13:31:08.253Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:31:08.146328389 +0000 UTC m=+108.916239064 (1 nodes), 78.051142ms from start of poll, 78.571803ms from cluster start. Timeout at 2024-07-05 13:31:08.252950848 +0000 UTC m=+109.022861523, 106.622459ms after stability -2024-07-05T13:31:08.253Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:08.358Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:08.253121442 +0000 UTC m=+109.023032117 (1 nodes), 1.28µs from start of poll, 185.364856ms from cluster start. Timeout at 2024-07-05 13:31:08.358305832 +0000 UTC m=+109.128216507, 105.18439ms after stability -2024-07-05T13:31:08.375Z [ERROR] server-50c17783-5ed4-7593-8f8f-97489bf931aa: failed to append to logs: error="unable to store logs within log store, err: "some error"" -2024-07-05T13:31:08.375Z [WARN] server-50c17783-5ed4-7593-8f8f-97489bf931aa: failed to get previous log: previous-index=9 last-index=7 error="log not found" -2024-07-05T13:31:08.375Z [INFO] server-45ab51d5-5a7f-bedd-6b4f-119ffe74236b: aborting pipeline replication: peer="{Voter server-50c17783-5ed4-7593-8f8f-97489bf931aa 50c17783-5ed4-7593-8f8f-97489bf931aa}" -2024-07-05T13:31:08.383Z [INFO] server-45ab51d5-5a7f-bedd-6b4f-119ffe74236b: pipelining replication: peer="{Voter server-50c17783-5ed4-7593-8f8f-97489bf931aa 50c17783-5ed4-7593-8f8f-97489bf931aa}" -2024-07-05T13:31:08.383Z [INFO] server-50c17783-5ed4-7593-8f8f-97489bf931aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50c17783-5ed4-7593-8f8f-97489bf931aa Address:50c17783-5ed4-7593-8f8f-97489bf931aa} {Suffrage:Voter ID:server-45ab51d5-5a7f-bedd-6b4f-119ffe74236b Address:45ab51d5-5a7f-bedd-6b4f-119ffe74236b}]" -2024-07-05T13:31:08.384Z [INFO] server-45ab51d5-5a7f-bedd-6b4f-119ffe74236b: aborting pipeline replication: peer="{Voter server-50c17783-5ed4-7593-8f8f-97489bf931aa 50c17783-5ed4-7593-8f8f-97489bf931aa}" +2024-07-05T13:37:24.870Z [INFO] server-95ec3fa5-8f50-b74a-edbc-c0699e702db0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95ec3fa5-8f50-b74a-edbc-c0699e702db0 Address:95ec3fa5-8f50-b74a-edbc-c0699e702db0} {Suffrage:Voter ID:server-b81ef8f1-6877-a9c9-eb94-99b8cf458867 Address:b81ef8f1-6877-a9c9-eb94-99b8cf458867}]" +2024-07-05T13:37:24.870Z [INFO] server-95ec3fa5-8f50-b74a-edbc-c0699e702db0: entering follower state: follower="Node at 95ec3fa5-8f50-b74a-edbc-c0699e702db0 [Follower]" leader-address= leader-id= +2024-07-05T13:37:24.871Z [INFO] server-b81ef8f1-6877-a9c9-eb94-99b8cf458867: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95ec3fa5-8f50-b74a-edbc-c0699e702db0 Address:95ec3fa5-8f50-b74a-edbc-c0699e702db0} {Suffrage:Voter ID:server-b81ef8f1-6877-a9c9-eb94-99b8cf458867 Address:b81ef8f1-6877-a9c9-eb94-99b8cf458867}]" +2024-07-05T13:37:24.871Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:24.871Z [INFO] server-b81ef8f1-6877-a9c9-eb94-99b8cf458867: entering follower state: follower="Node at b81ef8f1-6877-a9c9-eb94-99b8cf458867 [Follower]" leader-address= leader-id= +2024-07-05T13:37:24.934Z [WARN] server-b81ef8f1-6877-a9c9-eb94-99b8cf458867: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:24.934Z [INFO] server-b81ef8f1-6877-a9c9-eb94-99b8cf458867: entering candidate state: node="Node at b81ef8f1-6877-a9c9-eb94-99b8cf458867 [Candidate]" term=2 +2024-07-05T13:37:24.935Z [INFO] server-b81ef8f1-6877-a9c9-eb94-99b8cf458867: election won: term=2 tally=2 +2024-07-05T13:37:24.935Z [INFO] server-b81ef8f1-6877-a9c9-eb94-99b8cf458867: entering leader state: leader="Node at b81ef8f1-6877-a9c9-eb94-99b8cf458867 [Leader]" +2024-07-05T13:37:24.935Z [INFO] server-b81ef8f1-6877-a9c9-eb94-99b8cf458867: added peer, starting replication: peer=server-95ec3fa5-8f50-b74a-edbc-c0699e702db0 +2024-07-05T13:37:24.936Z [INFO] server-b81ef8f1-6877-a9c9-eb94-99b8cf458867: pipelining replication: peer="{Voter server-95ec3fa5-8f50-b74a-edbc-c0699e702db0 95ec3fa5-8f50-b74a-edbc-c0699e702db0}" +2024-07-05T13:37:25.041Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:24.935569259 +0000 UTC m=+110.931490419 (1 nodes), 64.130204ms from start of poll, 65.507137ms from cluster start. Timeout at 2024-07-05 13:37:25.041027 +0000 UTC m=+111.036947869, 105.45745ms after stability +2024-07-05T13:37:25.041Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:25.148Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:25.041423076 +0000 UTC m=+111.037343945 (1 nodes), 4.375µs from start of poll, 171.360663ms from cluster start. Timeout at 2024-07-05 13:37:25.146468408 +0000 UTC m=+111.142389568, 105.045623ms after stability +2024-07-05T13:37:25.170Z [ERROR] server-95ec3fa5-8f50-b74a-edbc-c0699e702db0: failed to append to logs: error="unable to store logs within log store, err: "some error"" +2024-07-05T13:37:25.171Z [WARN] server-95ec3fa5-8f50-b74a-edbc-c0699e702db0: failed to get previous log: previous-index=9 last-index=7 error="log not found" +2024-07-05T13:37:25.171Z [INFO] server-b81ef8f1-6877-a9c9-eb94-99b8cf458867: aborting pipeline replication: peer="{Voter server-95ec3fa5-8f50-b74a-edbc-c0699e702db0 95ec3fa5-8f50-b74a-edbc-c0699e702db0}" +2024-07-05T13:37:25.182Z [INFO] server-b81ef8f1-6877-a9c9-eb94-99b8cf458867: pipelining replication: peer="{Voter server-95ec3fa5-8f50-b74a-edbc-c0699e702db0 95ec3fa5-8f50-b74a-edbc-c0699e702db0}" +2024-07-05T13:37:25.183Z [INFO] server-95ec3fa5-8f50-b74a-edbc-c0699e702db0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95ec3fa5-8f50-b74a-edbc-c0699e702db0 Address:95ec3fa5-8f50-b74a-edbc-c0699e702db0} {Suffrage:Voter ID:server-b81ef8f1-6877-a9c9-eb94-99b8cf458867 Address:b81ef8f1-6877-a9c9-eb94-99b8cf458867}]" +2024-07-05T13:37:25.183Z [INFO] server-b81ef8f1-6877-a9c9-eb94-99b8cf458867: aborting pipeline replication: peer="{Voter server-95ec3fa5-8f50-b74a-edbc-c0699e702db0 95ec3fa5-8f50-b74a-edbc-c0699e702db0}" --- PASS: TestRaft_CacheLogWithStoreError (0.32s) === RUN TestRaft_ReloadConfig -2024-07-05T13:31:08.385Z [INFO] server-3697cf16-56e8-d8ec-cc7c-d165187f4ffa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3697cf16-56e8-d8ec-cc7c-d165187f4ffa Address:3697cf16-56e8-d8ec-cc7c-d165187f4ffa}]" +2024-07-05T13:37:25.188Z [INFO] server-f8e86e86-aaca-04c7-5e2c-924716fc2db6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8e86e86-aaca-04c7-5e2c-924716fc2db6 Address:f8e86e86-aaca-04c7-5e2c-924716fc2db6}]" --- PASS: TestRaft_ReloadConfig (0.00s) === RUN TestRaft_ReloadConfigValidates -2024-07-05T13:31:08.386Z [INFO] server-00323f25-a0c4-729c-574d-f3ce748a8db7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00323f25-a0c4-729c-574d-f3ce748a8db7 Address:00323f25-a0c4-729c-574d-f3ce748a8db7}]" +2024-07-05T13:37:25.192Z [INFO] server-15d02273-5a76-37c2-28ba-cd0e411ded48: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15d02273-5a76-37c2-28ba-cd0e411ded48 Address:15d02273-5a76-37c2-28ba-cd0e411ded48}]" --- PASS: TestRaft_ReloadConfigValidates (0.00s) === RUN TestRaft_InstallSnapshot_InvalidPeers -2024-07-05T13:31:08.387Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)" +2024-07-05T13:37:25.193Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)" --- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s) === RUN TestRaft_VoteNotGranted_WhenNodeNotInCluster -2024-07-05T13:31:08.389Z [INFO] server-b9a18fcd-23ba-18cf-7699-33a4b1eca589: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9a18fcd-23ba-18cf-7699-33a4b1eca589 Address:b9a18fcd-23ba-18cf-7699-33a4b1eca589} {Suffrage:Voter ID:server-59f59362-1296-5a3b-9806-706bc4e8f4e9 Address:59f59362-1296-5a3b-9806-706bc4e8f4e9} {Suffrage:Voter ID:server-9b46e560-f994-f489-ba4c-4706325f2a86 Address:9b46e560-f994-f489-ba4c-4706325f2a86}]" -2024-07-05T13:31:08.390Z [INFO] server-59f59362-1296-5a3b-9806-706bc4e8f4e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9a18fcd-23ba-18cf-7699-33a4b1eca589 Address:b9a18fcd-23ba-18cf-7699-33a4b1eca589} {Suffrage:Voter ID:server-59f59362-1296-5a3b-9806-706bc4e8f4e9 Address:59f59362-1296-5a3b-9806-706bc4e8f4e9} {Suffrage:Voter ID:server-9b46e560-f994-f489-ba4c-4706325f2a86 Address:9b46e560-f994-f489-ba4c-4706325f2a86}]" -2024-07-05T13:31:08.390Z [INFO] server-9b46e560-f994-f489-ba4c-4706325f2a86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9a18fcd-23ba-18cf-7699-33a4b1eca589 Address:b9a18fcd-23ba-18cf-7699-33a4b1eca589} {Suffrage:Voter ID:server-59f59362-1296-5a3b-9806-706bc4e8f4e9 Address:59f59362-1296-5a3b-9806-706bc4e8f4e9} {Suffrage:Voter ID:server-9b46e560-f994-f489-ba4c-4706325f2a86 Address:9b46e560-f994-f489-ba4c-4706325f2a86}]" -2024-07-05T13:31:08.390Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:08.390Z [INFO] server-b9a18fcd-23ba-18cf-7699-33a4b1eca589: entering follower state: follower="Node at b9a18fcd-23ba-18cf-7699-33a4b1eca589 [Follower]" leader-address= leader-id= -2024-07-05T13:31:08.390Z [INFO] server-59f59362-1296-5a3b-9806-706bc4e8f4e9: entering follower state: follower="Node at 59f59362-1296-5a3b-9806-706bc4e8f4e9 [Follower]" leader-address= leader-id= -2024-07-05T13:31:08.390Z [INFO] server-9b46e560-f994-f489-ba4c-4706325f2a86: entering follower state: follower="Node at 9b46e560-f994-f489-ba4c-4706325f2a86 [Follower]" leader-address= leader-id= -2024-07-05T13:31:08.468Z [WARN] server-9b46e560-f994-f489-ba4c-4706325f2a86: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:08.468Z [INFO] server-9b46e560-f994-f489-ba4c-4706325f2a86: entering candidate state: node="Node at 9b46e560-f994-f489-ba4c-4706325f2a86 [Candidate]" term=2 -2024-07-05T13:31:08.468Z [INFO] server-9b46e560-f994-f489-ba4c-4706325f2a86: election won: term=2 tally=2 -2024-07-05T13:31:08.469Z [INFO] server-9b46e560-f994-f489-ba4c-4706325f2a86: entering leader state: leader="Node at 9b46e560-f994-f489-ba4c-4706325f2a86 [Leader]" -2024-07-05T13:31:08.469Z [INFO] server-9b46e560-f994-f489-ba4c-4706325f2a86: added peer, starting replication: peer=server-b9a18fcd-23ba-18cf-7699-33a4b1eca589 -2024-07-05T13:31:08.469Z [INFO] server-9b46e560-f994-f489-ba4c-4706325f2a86: added peer, starting replication: peer=server-59f59362-1296-5a3b-9806-706bc4e8f4e9 -2024-07-05T13:31:08.469Z [INFO] server-9b46e560-f994-f489-ba4c-4706325f2a86: pipelining replication: peer="{Voter server-59f59362-1296-5a3b-9806-706bc4e8f4e9 59f59362-1296-5a3b-9806-706bc4e8f4e9}" -2024-07-05T13:31:08.469Z [INFO] server-9b46e560-f994-f489-ba4c-4706325f2a86: pipelining replication: peer="{Voter server-b9a18fcd-23ba-18cf-7699-33a4b1eca589 b9a18fcd-23ba-18cf-7699-33a4b1eca589}" -2024-07-05T13:31:08.574Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:08.469570688 +0000 UTC m=+109.239481363 (1 nodes), 79.176542ms from start of poll, 79.868918ms from cluster start. Timeout at 2024-07-05 13:31:08.574786276 +0000 UTC m=+109.344696951, 105.215588ms after stability -2024-07-05T13:31:08.579Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:31:08.685Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:31:08.580026653 +0000 UTC m=+109.349937328 (2 nodes), 1.6µs from start of poll, 190.324883ms from cluster start. Timeout at 2024-07-05 13:31:08.685517631 +0000 UTC m=+109.455428346, 105.491018ms after stability -2024-07-05T13:31:08.685Z [INFO] server-9b46e560-f994-f489-ba4c-4706325f2a86: updating configuration: command=RemoveServer server-id=server-b9a18fcd-23ba-18cf-7699-33a4b1eca589 server-addr= servers="[{Suffrage:Voter ID:server-59f59362-1296-5a3b-9806-706bc4e8f4e9 Address:59f59362-1296-5a3b-9806-706bc4e8f4e9} {Suffrage:Voter ID:server-9b46e560-f994-f489-ba4c-4706325f2a86 Address:9b46e560-f994-f489-ba4c-4706325f2a86}]" -2024-07-05T13:31:08.685Z [INFO] server-9b46e560-f994-f489-ba4c-4706325f2a86: removed peer, stopping replication: peer=server-b9a18fcd-23ba-18cf-7699-33a4b1eca589 last-index=3 -2024-07-05T13:31:08.685Z [INFO] server-9b46e560-f994-f489-ba4c-4706325f2a86: aborting pipeline replication: peer="{Voter server-b9a18fcd-23ba-18cf-7699-33a4b1eca589 b9a18fcd-23ba-18cf-7699-33a4b1eca589}" -2024-07-05T13:31:08.756Z [WARN] server-b9a18fcd-23ba-18cf-7699-33a4b1eca589: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election -2024-07-05T13:31:08.796Z [ERROR] server-9b46e560-f994-f489-ba4c-4706325f2a86: failed to heartbeat to: peer=59f59362-1296-5a3b-9806-706bc4e8f4e9 backoff time=10ms error="failed to connect to peer: 59f59362-1296-5a3b-9806-706bc4e8f4e9" -2024-07-05T13:31:08.799Z [ERROR] server-9b46e560-f994-f489-ba4c-4706325f2a86: failed to pipeline appendEntries: peer="{Voter server-59f59362-1296-5a3b-9806-706bc4e8f4e9 59f59362-1296-5a3b-9806-706bc4e8f4e9}" error="append pipeline closed" -2024-07-05T13:31:08.799Z [INFO] server-9b46e560-f994-f489-ba4c-4706325f2a86: aborting pipeline replication: peer="{Voter server-59f59362-1296-5a3b-9806-706bc4e8f4e9 59f59362-1296-5a3b-9806-706bc4e8f4e9}" -2024-07-05T13:31:08.806Z [ERROR] server-9b46e560-f994-f489-ba4c-4706325f2a86: failed to appendEntries to: peer="{Voter server-59f59362-1296-5a3b-9806-706bc4e8f4e9 59f59362-1296-5a3b-9806-706bc4e8f4e9}" error="failed to connect to peer: 59f59362-1296-5a3b-9806-706bc4e8f4e9" -2024-07-05T13:31:08.812Z [ERROR] server-9b46e560-f994-f489-ba4c-4706325f2a86: failed to heartbeat to: peer=59f59362-1296-5a3b-9806-706bc4e8f4e9 backoff time=10ms error="failed to connect to peer: 59f59362-1296-5a3b-9806-706bc4e8f4e9" -2024-07-05T13:31:08.825Z [ERROR] server-9b46e560-f994-f489-ba4c-4706325f2a86: failed to appendEntries to: peer="{Voter server-59f59362-1296-5a3b-9806-706bc4e8f4e9 59f59362-1296-5a3b-9806-706bc4e8f4e9}" error="failed to connect to peer: 59f59362-1296-5a3b-9806-706bc4e8f4e9" -2024-07-05T13:31:08.830Z [ERROR] server-9b46e560-f994-f489-ba4c-4706325f2a86: failed to heartbeat to: peer=59f59362-1296-5a3b-9806-706bc4e8f4e9 backoff time=10ms error="failed to connect to peer: 59f59362-1296-5a3b-9806-706bc4e8f4e9" -2024-07-05T13:31:08.840Z [WARN] server-9b46e560-f994-f489-ba4c-4706325f2a86: failed to contact: server-id=server-59f59362-1296-5a3b-9806-706bc4e8f4e9 time=50.515548ms -2024-07-05T13:31:08.840Z [WARN] server-9b46e560-f994-f489-ba4c-4706325f2a86: failed to contact quorum of nodes, stepping down -2024-07-05T13:31:08.841Z [INFO] server-9b46e560-f994-f489-ba4c-4706325f2a86: entering follower state: follower="Node at 9b46e560-f994-f489-ba4c-4706325f2a86 [Follower]" leader-address= leader-id= -2024-07-05T13:31:08.842Z [ERROR] server-9b46e560-f994-f489-ba4c-4706325f2a86: failed to appendEntries to: peer="{Voter server-59f59362-1296-5a3b-9806-706bc4e8f4e9 59f59362-1296-5a3b-9806-706bc4e8f4e9}" error="failed to connect to peer: 59f59362-1296-5a3b-9806-706bc4e8f4e9" -2024-07-05T13:31:08.896Z [WARN] server-59f59362-1296-5a3b-9806-706bc4e8f4e9: heartbeat timeout reached, starting election: last-leader-addr=9b46e560-f994-f489-ba4c-4706325f2a86 last-leader-id=server-9b46e560-f994-f489-ba4c-4706325f2a86 -2024-07-05T13:31:08.896Z [INFO] server-59f59362-1296-5a3b-9806-706bc4e8f4e9: entering candidate state: node="Node at 59f59362-1296-5a3b-9806-706bc4e8f4e9 [Candidate]" term=3 -2024-07-05T13:31:08.896Z [ERROR] server-59f59362-1296-5a3b-9806-706bc4e8f4e9: failed to make requestVote RPC: target="{Voter server-9b46e560-f994-f489-ba4c-4706325f2a86 9b46e560-f994-f489-ba4c-4706325f2a86}" error="failed to connect to peer: 9b46e560-f994-f489-ba4c-4706325f2a86" term=3 -2024-07-05T13:31:08.897Z [WARN] server-59f59362-1296-5a3b-9806-706bc4e8f4e9: rejecting vote request since node is not in configuration: from=b9a18fcd-23ba-18cf-7699-33a4b1eca589 ---- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.51s) +2024-07-05T13:37:25.197Z [INFO] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771 Address:ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771} {Suffrage:Voter ID:server-9046b03f-7be4-ae72-8e66-4971c60b6d84 Address:9046b03f-7be4-ae72-8e66-4971c60b6d84} {Suffrage:Voter ID:server-f6ac213d-5c47-22e5-0409-01678fff16b3 Address:f6ac213d-5c47-22e5-0409-01678fff16b3}]" +2024-07-05T13:37:25.197Z [INFO] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: entering follower state: follower="Node at ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771 [Follower]" leader-address= leader-id= +2024-07-05T13:37:25.198Z [INFO] server-9046b03f-7be4-ae72-8e66-4971c60b6d84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771 Address:ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771} {Suffrage:Voter ID:server-9046b03f-7be4-ae72-8e66-4971c60b6d84 Address:9046b03f-7be4-ae72-8e66-4971c60b6d84} {Suffrage:Voter ID:server-f6ac213d-5c47-22e5-0409-01678fff16b3 Address:f6ac213d-5c47-22e5-0409-01678fff16b3}]" +2024-07-05T13:37:25.198Z [INFO] server-9046b03f-7be4-ae72-8e66-4971c60b6d84: entering follower state: follower="Node at 9046b03f-7be4-ae72-8e66-4971c60b6d84 [Follower]" leader-address= leader-id= +2024-07-05T13:37:25.198Z [INFO] server-f6ac213d-5c47-22e5-0409-01678fff16b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771 Address:ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771} {Suffrage:Voter ID:server-9046b03f-7be4-ae72-8e66-4971c60b6d84 Address:9046b03f-7be4-ae72-8e66-4971c60b6d84} {Suffrage:Voter ID:server-f6ac213d-5c47-22e5-0409-01678fff16b3 Address:f6ac213d-5c47-22e5-0409-01678fff16b3}]" +2024-07-05T13:37:25.198Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:25.199Z [INFO] server-f6ac213d-5c47-22e5-0409-01678fff16b3: entering follower state: follower="Node at f6ac213d-5c47-22e5-0409-01678fff16b3 [Follower]" leader-address= leader-id= +2024-07-05T13:37:25.254Z [WARN] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:25.254Z [INFO] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: entering candidate state: node="Node at ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771 [Candidate]" term=2 +2024-07-05T13:37:25.254Z [INFO] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: election won: term=2 tally=2 +2024-07-05T13:37:25.254Z [INFO] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: entering leader state: leader="Node at ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771 [Leader]" +2024-07-05T13:37:25.254Z [INFO] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: added peer, starting replication: peer=server-9046b03f-7be4-ae72-8e66-4971c60b6d84 +2024-07-05T13:37:25.254Z [INFO] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: added peer, starting replication: peer=server-f6ac213d-5c47-22e5-0409-01678fff16b3 +2024-07-05T13:37:25.255Z [INFO] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: pipelining replication: peer="{Voter server-9046b03f-7be4-ae72-8e66-4971c60b6d84 9046b03f-7be4-ae72-8e66-4971c60b6d84}" +2024-07-05T13:37:25.255Z [INFO] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: pipelining replication: peer="{Voter server-f6ac213d-5c47-22e5-0409-01678fff16b3 f6ac213d-5c47-22e5-0409-01678fff16b3}" +2024-07-05T13:37:25.360Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:25.254928511 +0000 UTC m=+111.250849671 (1 nodes), 55.95202ms from start of poll, 57.590865ms from cluster start. Timeout at 2024-07-05 13:37:25.360069799 +0000 UTC m=+111.355990668, 105.140997ms after stability +2024-07-05T13:37:25.365Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:25.471Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:25.365632365 +0000 UTC m=+111.361553525 (2 nodes), 5.834µs from start of poll, 168.294719ms from cluster start. Timeout at 2024-07-05 13:37:25.470696654 +0000 UTC m=+111.466618106, 105.064581ms after stability +2024-07-05T13:37:25.471Z [INFO] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: updating configuration: command=RemoveServer server-id=server-9046b03f-7be4-ae72-8e66-4971c60b6d84 server-addr= servers="[{Suffrage:Voter ID:server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771 Address:ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771} {Suffrage:Voter ID:server-f6ac213d-5c47-22e5-0409-01678fff16b3 Address:f6ac213d-5c47-22e5-0409-01678fff16b3}]" +2024-07-05T13:37:25.471Z [INFO] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: removed peer, stopping replication: peer=server-9046b03f-7be4-ae72-8e66-4971c60b6d84 last-index=3 +2024-07-05T13:37:25.471Z [INFO] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: aborting pipeline replication: peer="{Voter server-9046b03f-7be4-ae72-8e66-4971c60b6d84 9046b03f-7be4-ae72-8e66-4971c60b6d84}" +2024-07-05T13:37:25.564Z [WARN] server-9046b03f-7be4-ae72-8e66-4971c60b6d84: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election +2024-07-05T13:37:25.583Z [ERROR] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: failed to heartbeat to: peer=f6ac213d-5c47-22e5-0409-01678fff16b3 backoff time=10ms error="failed to connect to peer: f6ac213d-5c47-22e5-0409-01678fff16b3" +2024-07-05T13:37:25.583Z [ERROR] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: failed to pipeline appendEntries: peer="{Voter server-f6ac213d-5c47-22e5-0409-01678fff16b3 f6ac213d-5c47-22e5-0409-01678fff16b3}" error="append pipeline closed" +2024-07-05T13:37:25.584Z [INFO] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: aborting pipeline replication: peer="{Voter server-f6ac213d-5c47-22e5-0409-01678fff16b3 f6ac213d-5c47-22e5-0409-01678fff16b3}" +2024-07-05T13:37:25.594Z [ERROR] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: failed to appendEntries to: peer="{Voter server-f6ac213d-5c47-22e5-0409-01678fff16b3 f6ac213d-5c47-22e5-0409-01678fff16b3}" error="failed to connect to peer: f6ac213d-5c47-22e5-0409-01678fff16b3" +2024-07-05T13:37:25.612Z [ERROR] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: failed to heartbeat to: peer=f6ac213d-5c47-22e5-0409-01678fff16b3 backoff time=10ms error="failed to connect to peer: f6ac213d-5c47-22e5-0409-01678fff16b3" +2024-07-05T13:37:25.616Z [ERROR] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: failed to appendEntries to: peer="{Voter server-f6ac213d-5c47-22e5-0409-01678fff16b3 f6ac213d-5c47-22e5-0409-01678fff16b3}" error="failed to connect to peer: f6ac213d-5c47-22e5-0409-01678fff16b3" +2024-07-05T13:37:25.627Z [WARN] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: failed to contact: server-id=server-f6ac213d-5c47-22e5-0409-01678fff16b3 time=50.187626ms +2024-07-05T13:37:25.627Z [WARN] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: failed to contact quorum of nodes, stepping down +2024-07-05T13:37:25.627Z [INFO] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: entering follower state: follower="Node at ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771 [Follower]" leader-address= leader-id= +2024-07-05T13:37:25.634Z [ERROR] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: failed to heartbeat to: peer=f6ac213d-5c47-22e5-0409-01678fff16b3 backoff time=10ms error="failed to connect to peer: f6ac213d-5c47-22e5-0409-01678fff16b3" +2024-07-05T13:37:25.636Z [ERROR] server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771: failed to appendEntries to: peer="{Voter server-f6ac213d-5c47-22e5-0409-01678fff16b3 f6ac213d-5c47-22e5-0409-01678fff16b3}" error="failed to connect to peer: f6ac213d-5c47-22e5-0409-01678fff16b3" +2024-07-05T13:37:25.661Z [WARN] server-f6ac213d-5c47-22e5-0409-01678fff16b3: heartbeat timeout reached, starting election: last-leader-addr=ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771 last-leader-id=server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771 +2024-07-05T13:37:25.661Z [INFO] server-f6ac213d-5c47-22e5-0409-01678fff16b3: entering candidate state: node="Node at f6ac213d-5c47-22e5-0409-01678fff16b3 [Candidate]" term=3 +2024-07-05T13:37:25.661Z [ERROR] server-f6ac213d-5c47-22e5-0409-01678fff16b3: failed to make requestVote RPC: target="{Voter server-ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771 ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771}" error="failed to connect to peer: ae3bcfa6-0488-ec3f-8bc3-393bcb3cc771" term=3 +2024-07-05T13:37:25.683Z [WARN] server-f6ac213d-5c47-22e5-0409-01678fff16b3: rejecting vote request since node is not in configuration: from=9046b03f-7be4-ae72-8e66-4971c60b6d84 +--- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.49s) === RUN TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin -2024-07-05T13:31:08.899Z [INFO] server-8c4c9e3d-f629-5103-837f-faa8fd851fe9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 Address:8c4c9e3d-f629-5103-837f-faa8fd851fe9} {Suffrage:Voter ID:server-4b7e7d48-e59c-819f-a827-33dec821bb60 Address:4b7e7d48-e59c-819f-a827-33dec821bb60} {Suffrage:Voter ID:server-1e2cbf34-46c8-a9d2-245a-9d83fe5441f4 Address:1e2cbf34-46c8-a9d2-245a-9d83fe5441f4}]" -2024-07-05T13:31:08.899Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 Address:8c4c9e3d-f629-5103-837f-faa8fd851fe9} {Suffrage:Voter ID:server-4b7e7d48-e59c-819f-a827-33dec821bb60 Address:4b7e7d48-e59c-819f-a827-33dec821bb60} {Suffrage:Voter ID:server-1e2cbf34-46c8-a9d2-245a-9d83fe5441f4 Address:1e2cbf34-46c8-a9d2-245a-9d83fe5441f4}]" -2024-07-05T13:31:08.899Z [INFO] server-1e2cbf34-46c8-a9d2-245a-9d83fe5441f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 Address:8c4c9e3d-f629-5103-837f-faa8fd851fe9} {Suffrage:Voter ID:server-4b7e7d48-e59c-819f-a827-33dec821bb60 Address:4b7e7d48-e59c-819f-a827-33dec821bb60} {Suffrage:Voter ID:server-1e2cbf34-46c8-a9d2-245a-9d83fe5441f4 Address:1e2cbf34-46c8-a9d2-245a-9d83fe5441f4}]" -2024-07-05T13:31:08.899Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:08.899Z [INFO] server-8c4c9e3d-f629-5103-837f-faa8fd851fe9: entering follower state: follower="Node at 8c4c9e3d-f629-5103-837f-faa8fd851fe9 [Follower]" leader-address= leader-id= -2024-07-05T13:31:08.900Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: entering follower state: follower="Node at 4b7e7d48-e59c-819f-a827-33dec821bb60 [Follower]" leader-address= leader-id= -2024-07-05T13:31:08.900Z [INFO] server-1e2cbf34-46c8-a9d2-245a-9d83fe5441f4: entering follower state: follower="Node at 1e2cbf34-46c8-a9d2-245a-9d83fe5441f4 [Follower]" leader-address= leader-id= -2024-07-05T13:31:08.951Z [WARN] server-4b7e7d48-e59c-819f-a827-33dec821bb60: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:08.951Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: entering candidate state: node="Node at 4b7e7d48-e59c-819f-a827-33dec821bb60 [Candidate]" term=2 -2024-07-05T13:31:08.951Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: election won: term=2 tally=2 -2024-07-05T13:31:08.951Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: entering leader state: leader="Node at 4b7e7d48-e59c-819f-a827-33dec821bb60 [Leader]" -2024-07-05T13:31:08.951Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: added peer, starting replication: peer=server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 -2024-07-05T13:31:08.951Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: added peer, starting replication: peer=server-1e2cbf34-46c8-a9d2-245a-9d83fe5441f4 -2024-07-05T13:31:08.952Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: pipelining replication: peer="{Voter server-1e2cbf34-46c8-a9d2-245a-9d83fe5441f4 1e2cbf34-46c8-a9d2-245a-9d83fe5441f4}" -2024-07-05T13:31:08.953Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: pipelining replication: peer="{Voter server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 8c4c9e3d-f629-5103-837f-faa8fd851fe9}" -2024-07-05T13:31:09.058Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:08.952877011 +0000 UTC m=+109.722787686 (1 nodes), 53.043819ms from start of poll, 53.677077ms from cluster start. Timeout at 2024-07-05 13:31:09.058795295 +0000 UTC m=+109.828705970, 105.918284ms after stability -2024-07-05T13:31:09.065Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:31:09.170Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:31:09.065445542 +0000 UTC m=+109.835356217 (2 nodes), 2.16µs from start of poll, 166.245608ms from cluster start. Timeout at 2024-07-05 13:31:09.170488816 +0000 UTC m=+109.940399491, 105.043274ms after stability -2024-07-05T13:31:09.173Z [ERROR] server-4b7e7d48-e59c-819f-a827-33dec821bb60: failed to heartbeat to: peer=8c4c9e3d-f629-5103-837f-faa8fd851fe9 backoff time=10ms error="failed to connect to peer: 8c4c9e3d-f629-5103-837f-faa8fd851fe9" -2024-07-05T13:31:09.175Z [ERROR] server-4b7e7d48-e59c-819f-a827-33dec821bb60: failed to pipeline appendEntries: peer="{Voter server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 8c4c9e3d-f629-5103-837f-faa8fd851fe9}" error="append pipeline closed" -2024-07-05T13:31:09.175Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: aborting pipeline replication: peer="{Voter server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 8c4c9e3d-f629-5103-837f-faa8fd851fe9}" -2024-07-05T13:31:09.183Z [ERROR] server-4b7e7d48-e59c-819f-a827-33dec821bb60: failed to appendEntries to: peer="{Voter server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 8c4c9e3d-f629-5103-837f-faa8fd851fe9}" error="failed to connect to peer: 8c4c9e3d-f629-5103-837f-faa8fd851fe9" -2024-07-05T13:31:09.193Z [ERROR] server-4b7e7d48-e59c-819f-a827-33dec821bb60: failed to heartbeat to: peer=8c4c9e3d-f629-5103-837f-faa8fd851fe9 backoff time=10ms error="failed to connect to peer: 8c4c9e3d-f629-5103-837f-faa8fd851fe9" -2024-07-05T13:31:09.204Z [ERROR] server-4b7e7d48-e59c-819f-a827-33dec821bb60: failed to appendEntries to: peer="{Voter server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 8c4c9e3d-f629-5103-837f-faa8fd851fe9}" error="failed to connect to peer: 8c4c9e3d-f629-5103-837f-faa8fd851fe9" -2024-07-05T13:31:09.214Z [ERROR] server-4b7e7d48-e59c-819f-a827-33dec821bb60: failed to heartbeat to: peer=8c4c9e3d-f629-5103-837f-faa8fd851fe9 backoff time=10ms error="failed to connect to peer: 8c4c9e3d-f629-5103-837f-faa8fd851fe9" -2024-07-05T13:31:09.222Z [WARN] server-4b7e7d48-e59c-819f-a827-33dec821bb60: failed to contact: server-id=server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 time=53.343608ms -2024-07-05T13:31:09.223Z [ERROR] server-4b7e7d48-e59c-819f-a827-33dec821bb60: failed to appendEntries to: peer="{Voter server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 8c4c9e3d-f629-5103-837f-faa8fd851fe9}" error="failed to connect to peer: 8c4c9e3d-f629-5103-837f-faa8fd851fe9" -2024-07-05T13:31:09.233Z [ERROR] server-4b7e7d48-e59c-819f-a827-33dec821bb60: failed to heartbeat to: peer=8c4c9e3d-f629-5103-837f-faa8fd851fe9 backoff time=20ms error="failed to connect to peer: 8c4c9e3d-f629-5103-837f-faa8fd851fe9" -2024-07-05T13:31:09.238Z [WARN] server-8c4c9e3d-f629-5103-837f-faa8fd851fe9: heartbeat timeout reached, starting election: last-leader-addr=4b7e7d48-e59c-819f-a827-33dec821bb60 last-leader-id=server-4b7e7d48-e59c-819f-a827-33dec821bb60 -2024-07-05T13:31:09.238Z [INFO] server-8c4c9e3d-f629-5103-837f-faa8fd851fe9: entering candidate state: node="Node at 8c4c9e3d-f629-5103-837f-faa8fd851fe9 [Candidate]" term=3 -2024-07-05T13:31:09.238Z [ERROR] server-8c4c9e3d-f629-5103-837f-faa8fd851fe9: failed to make requestVote RPC: target="{Voter server-1e2cbf34-46c8-a9d2-245a-9d83fe5441f4 1e2cbf34-46c8-a9d2-245a-9d83fe5441f4}" error="failed to connect to peer: 1e2cbf34-46c8-a9d2-245a-9d83fe5441f4" term=3 -2024-07-05T13:31:09.238Z [ERROR] server-8c4c9e3d-f629-5103-837f-faa8fd851fe9: failed to make requestVote RPC: target="{Voter server-4b7e7d48-e59c-819f-a827-33dec821bb60 4b7e7d48-e59c-819f-a827-33dec821bb60}" error="failed to connect to peer: 4b7e7d48-e59c-819f-a827-33dec821bb60" term=3 -2024-07-05T13:31:09.250Z [ERROR] server-4b7e7d48-e59c-819f-a827-33dec821bb60: failed to appendEntries to: peer="{Voter server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 8c4c9e3d-f629-5103-837f-faa8fd851fe9}" error="failed to connect to peer: 8c4c9e3d-f629-5103-837f-faa8fd851fe9" -2024-07-05T13:31:09.260Z [ERROR] server-4b7e7d48-e59c-819f-a827-33dec821bb60: failed to heartbeat to: peer=8c4c9e3d-f629-5103-837f-faa8fd851fe9 backoff time=25ms error="failed to connect to peer: 8c4c9e3d-f629-5103-837f-faa8fd851fe9" -2024-07-05T13:31:09.267Z [WARN] server-4b7e7d48-e59c-819f-a827-33dec821bb60: failed to contact: server-id=server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 time=98.243473ms -2024-07-05T13:31:09.275Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: updating configuration: command=RemoveServer server-id=server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 server-addr= servers="[{Suffrage:Voter ID:server-4b7e7d48-e59c-819f-a827-33dec821bb60 Address:4b7e7d48-e59c-819f-a827-33dec821bb60} {Suffrage:Voter ID:server-1e2cbf34-46c8-a9d2-245a-9d83fe5441f4 Address:1e2cbf34-46c8-a9d2-245a-9d83fe5441f4}]" -2024-07-05T13:31:09.275Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: removed peer, stopping replication: peer=server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 last-index=3 -2024-07-05T13:31:09.276Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: updating configuration: command=AddNonvoter server-id=server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 server-addr=8c4c9e3d-f629-5103-837f-faa8fd851fe9 servers="[{Suffrage:Voter ID:server-4b7e7d48-e59c-819f-a827-33dec821bb60 Address:4b7e7d48-e59c-819f-a827-33dec821bb60} {Suffrage:Voter ID:server-1e2cbf34-46c8-a9d2-245a-9d83fe5441f4 Address:1e2cbf34-46c8-a9d2-245a-9d83fe5441f4} {Suffrage:Nonvoter ID:server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 Address:8c4c9e3d-f629-5103-837f-faa8fd851fe9}]" -2024-07-05T13:31:09.277Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: added peer, starting replication: peer=server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 -2024-07-05T13:31:09.277Z [ERROR] server-4b7e7d48-e59c-819f-a827-33dec821bb60: failed to appendEntries to: peer="{Nonvoter server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 8c4c9e3d-f629-5103-837f-faa8fd851fe9}" error="failed to connect to peer: 8c4c9e3d-f629-5103-837f-faa8fd851fe9" -2024-07-05T13:31:09.294Z [ERROR] server-4b7e7d48-e59c-819f-a827-33dec821bb60: peer has newer term, stopping replication: peer="{Nonvoter server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 8c4c9e3d-f629-5103-837f-faa8fd851fe9}" -2024-07-05T13:31:09.294Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: entering follower state: follower="Node at 4b7e7d48-e59c-819f-a827-33dec821bb60 [Follower]" leader-address= leader-id= -2024-07-05T13:31:09.294Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: aborting pipeline replication: peer="{Voter server-1e2cbf34-46c8-a9d2-245a-9d83fe5441f4 1e2cbf34-46c8-a9d2-245a-9d83fe5441f4}" -2024-07-05T13:31:09.299Z [ERROR] server-4b7e7d48-e59c-819f-a827-33dec821bb60: peer has newer term, stopping replication: peer="{Voter server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 8c4c9e3d-f629-5103-837f-faa8fd851fe9}" -2024-07-05T13:31:09.318Z [WARN] server-8c4c9e3d-f629-5103-837f-faa8fd851fe9: Election timeout reached, restarting election -2024-07-05T13:31:09.318Z [INFO] server-8c4c9e3d-f629-5103-837f-faa8fd851fe9: entering candidate state: node="Node at 8c4c9e3d-f629-5103-837f-faa8fd851fe9 [Candidate]" term=23 -2024-07-05T13:31:09.318Z [WARN] server-1e2cbf34-46c8-a9d2-245a-9d83fe5441f4: rejecting vote request since we have a leader: from=8c4c9e3d-f629-5103-837f-faa8fd851fe9 leader=4b7e7d48-e59c-819f-a827-33dec821bb60 leader-id=server-4b7e7d48-e59c-819f-a827-33dec821bb60 -2024-07-05T13:31:09.318Z [WARN] server-4b7e7d48-e59c-819f-a827-33dec821bb60: rejecting vote request since node is not a voter: from=8c4c9e3d-f629-5103-837f-faa8fd851fe9 -2024-07-05T13:31:09.383Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:09.396Z [WARN] server-4b7e7d48-e59c-819f-a827-33dec821bb60: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:09.396Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: entering candidate state: node="Node at 4b7e7d48-e59c-819f-a827-33dec821bb60 [Candidate]" term=24 -2024-07-05T13:31:09.396Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: election won: term=24 tally=2 -2024-07-05T13:31:09.396Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: entering leader state: leader="Node at 4b7e7d48-e59c-819f-a827-33dec821bb60 [Leader]" -2024-07-05T13:31:09.396Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: added peer, starting replication: peer=server-1e2cbf34-46c8-a9d2-245a-9d83fe5441f4 -2024-07-05T13:31:09.396Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: added peer, starting replication: peer=server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 -2024-07-05T13:31:09.398Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: pipelining replication: peer="{Voter server-1e2cbf34-46c8-a9d2-245a-9d83fe5441f4 1e2cbf34-46c8-a9d2-245a-9d83fe5441f4}" -2024-07-05T13:31:09.399Z [WARN] server-8c4c9e3d-f629-5103-837f-faa8fd851fe9: failed to get previous log: previous-index=4 last-index=2 error="log not found" -2024-07-05T13:31:09.399Z [INFO] server-8c4c9e3d-f629-5103-837f-faa8fd851fe9: entering follower state: follower="Node at 8c4c9e3d-f629-5103-837f-faa8fd851fe9 [Follower]" leader-address=4b7e7d48-e59c-819f-a827-33dec821bb60 leader-id=server-4b7e7d48-e59c-819f-a827-33dec821bb60 -2024-07-05T13:31:09.399Z [WARN] server-4b7e7d48-e59c-819f-a827-33dec821bb60: appendEntries rejected, sending older logs: peer="{Nonvoter server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 8c4c9e3d-f629-5103-837f-faa8fd851fe9}" next=3 -2024-07-05T13:31:09.400Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: pipelining replication: peer="{Nonvoter server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 8c4c9e3d-f629-5103-837f-faa8fd851fe9}" -2024-07-05T13:31:09.505Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:09.400094681 +0000 UTC m=+110.170005356 (1 nodes), 16.703174ms from start of poll, 500.894747ms from cluster start. Timeout at 2024-07-05 13:31:09.505359947 +0000 UTC m=+110.275270622, 105.265266ms after stability -2024-07-05T13:31:09.553Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: updating configuration: command=RemoveServer server-id=server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 server-addr= servers="[{Suffrage:Voter ID:server-4b7e7d48-e59c-819f-a827-33dec821bb60 Address:4b7e7d48-e59c-819f-a827-33dec821bb60} {Suffrage:Voter ID:server-1e2cbf34-46c8-a9d2-245a-9d83fe5441f4 Address:1e2cbf34-46c8-a9d2-245a-9d83fe5441f4}]" -2024-07-05T13:31:09.553Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: removed peer, stopping replication: peer=server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 last-index=106 -2024-07-05T13:31:09.553Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: aborting pipeline replication: peer="{Nonvoter server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 8c4c9e3d-f629-5103-837f-faa8fd851fe9}" -2024-07-05T13:31:09.553Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: updating configuration: command=AddVoter server-id=server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 server-addr=8c4c9e3d-f629-5103-837f-faa8fd851fe9 servers="[{Suffrage:Voter ID:server-4b7e7d48-e59c-819f-a827-33dec821bb60 Address:4b7e7d48-e59c-819f-a827-33dec821bb60} {Suffrage:Voter ID:server-1e2cbf34-46c8-a9d2-245a-9d83fe5441f4 Address:1e2cbf34-46c8-a9d2-245a-9d83fe5441f4} {Suffrage:Voter ID:server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 Address:8c4c9e3d-f629-5103-837f-faa8fd851fe9}]" -2024-07-05T13:31:09.553Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: added peer, starting replication: peer=server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 -2024-07-05T13:31:09.553Z [WARN] server-8c4c9e3d-f629-5103-837f-faa8fd851fe9: failed to get previous log: previous-index=107 last-index=106 error="log not found" -2024-07-05T13:31:09.553Z [WARN] server-4b7e7d48-e59c-819f-a827-33dec821bb60: appendEntries rejected, sending older logs: peer="{Voter server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 8c4c9e3d-f629-5103-837f-faa8fd851fe9}" next=107 -2024-07-05T13:31:09.554Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: pipelining replication: peer="{Voter server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 8c4c9e3d-f629-5103-837f-faa8fd851fe9}" -2024-07-05T13:31:10.660Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:10.766Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:10.660592336 +0000 UTC m=+111.430503011 (1 nodes), 2.6µs from start of poll, 1.761392402s from cluster start. Timeout at 2024-07-05 13:31:10.766434983 +0000 UTC m=+111.536345658, 105.842647ms after stability -2024-07-05T13:31:10.766Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: aborting pipeline replication: peer="{Voter server-8c4c9e3d-f629-5103-837f-faa8fd851fe9 8c4c9e3d-f629-5103-837f-faa8fd851fe9}" -2024-07-05T13:31:10.766Z [INFO] server-4b7e7d48-e59c-819f-a827-33dec821bb60: aborting pipeline replication: peer="{Voter server-1e2cbf34-46c8-a9d2-245a-9d83fe5441f4 1e2cbf34-46c8-a9d2-245a-9d83fe5441f4}" ---- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.88s) +2024-07-05T13:37:25.691Z [INFO] server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf Address:8b34a5c7-035a-c25e-2c72-eaeccaffe1cf} {Suffrage:Voter ID:server-67170892-8a08-03c9-fd0c-1606d5dd00be Address:67170892-8a08-03c9-fd0c-1606d5dd00be} {Suffrage:Voter ID:server-bb065309-aa4f-1089-caf8-d2abe062b8c3 Address:bb065309-aa4f-1089-caf8-d2abe062b8c3}]" +2024-07-05T13:37:25.691Z [INFO] server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf: entering follower state: follower="Node at 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf [Follower]" leader-address= leader-id= +2024-07-05T13:37:25.692Z [INFO] server-67170892-8a08-03c9-fd0c-1606d5dd00be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf Address:8b34a5c7-035a-c25e-2c72-eaeccaffe1cf} {Suffrage:Voter ID:server-67170892-8a08-03c9-fd0c-1606d5dd00be Address:67170892-8a08-03c9-fd0c-1606d5dd00be} {Suffrage:Voter ID:server-bb065309-aa4f-1089-caf8-d2abe062b8c3 Address:bb065309-aa4f-1089-caf8-d2abe062b8c3}]" +2024-07-05T13:37:25.692Z [INFO] server-67170892-8a08-03c9-fd0c-1606d5dd00be: entering follower state: follower="Node at 67170892-8a08-03c9-fd0c-1606d5dd00be [Follower]" leader-address= leader-id= +2024-07-05T13:37:25.692Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf Address:8b34a5c7-035a-c25e-2c72-eaeccaffe1cf} {Suffrage:Voter ID:server-67170892-8a08-03c9-fd0c-1606d5dd00be Address:67170892-8a08-03c9-fd0c-1606d5dd00be} {Suffrage:Voter ID:server-bb065309-aa4f-1089-caf8-d2abe062b8c3 Address:bb065309-aa4f-1089-caf8-d2abe062b8c3}]" +2024-07-05T13:37:25.693Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:25.693Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: entering follower state: follower="Node at bb065309-aa4f-1089-caf8-d2abe062b8c3 [Follower]" leader-address= leader-id= +2024-07-05T13:37:25.749Z [WARN] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:25.749Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: entering candidate state: node="Node at bb065309-aa4f-1089-caf8-d2abe062b8c3 [Candidate]" term=2 +2024-07-05T13:37:25.749Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: election won: term=2 tally=2 +2024-07-05T13:37:25.749Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: entering leader state: leader="Node at bb065309-aa4f-1089-caf8-d2abe062b8c3 [Leader]" +2024-07-05T13:37:25.749Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: added peer, starting replication: peer=server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf +2024-07-05T13:37:25.749Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: added peer, starting replication: peer=server-67170892-8a08-03c9-fd0c-1606d5dd00be +2024-07-05T13:37:25.750Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: pipelining replication: peer="{Voter server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf}" +2024-07-05T13:37:25.750Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: pipelining replication: peer="{Voter server-67170892-8a08-03c9-fd0c-1606d5dd00be 67170892-8a08-03c9-fd0c-1606d5dd00be}" +2024-07-05T13:37:25.855Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:25.749774558 +0000 UTC m=+111.745695718 (1 nodes), 56.472052ms from start of poll, 58.75226ms from cluster start. Timeout at 2024-07-05 13:37:25.854888431 +0000 UTC m=+111.850809882, 105.114164ms after stability +2024-07-05T13:37:25.860Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:25.966Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:25.86049387 +0000 UTC m=+111.856414738 (2 nodes), 5.833µs from start of poll, 169.47128ms from cluster start. Timeout at 2024-07-05 13:37:25.966454143 +0000 UTC m=+111.962375012, 105.960274ms after stability +2024-07-05T13:37:25.969Z [ERROR] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: failed to pipeline appendEntries: peer="{Voter server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf}" error="append pipeline closed" +2024-07-05T13:37:25.969Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: aborting pipeline replication: peer="{Voter server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf}" +2024-07-05T13:37:25.972Z [ERROR] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: failed to heartbeat to: peer=8b34a5c7-035a-c25e-2c72-eaeccaffe1cf backoff time=10ms error="failed to connect to peer: 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf" +2024-07-05T13:37:25.976Z [ERROR] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: failed to appendEntries to: peer="{Voter server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf}" error="failed to connect to peer: 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf" +2024-07-05T13:37:25.990Z [ERROR] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: failed to heartbeat to: peer=8b34a5c7-035a-c25e-2c72-eaeccaffe1cf backoff time=10ms error="failed to connect to peer: 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf" +2024-07-05T13:37:25.994Z [ERROR] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: failed to appendEntries to: peer="{Voter server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf}" error="failed to connect to peer: 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf" +2024-07-05T13:37:26.009Z [ERROR] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: failed to heartbeat to: peer=8b34a5c7-035a-c25e-2c72-eaeccaffe1cf backoff time=10ms error="failed to connect to peer: 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf" +2024-07-05T13:37:26.012Z [ERROR] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: failed to appendEntries to: peer="{Voter server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf}" error="failed to connect to peer: 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf" +2024-07-05T13:37:26.014Z [WARN] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: failed to contact: server-id=server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf time=50.1885ms +2024-07-05T13:37:26.027Z [ERROR] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: failed to heartbeat to: peer=8b34a5c7-035a-c25e-2c72-eaeccaffe1cf backoff time=20ms error="failed to connect to peer: 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf" +2024-07-05T13:37:26.038Z [WARN] server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf: heartbeat timeout reached, starting election: last-leader-addr=bb065309-aa4f-1089-caf8-d2abe062b8c3 last-leader-id=server-bb065309-aa4f-1089-caf8-d2abe062b8c3 +2024-07-05T13:37:26.038Z [INFO] server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf: entering candidate state: node="Node at 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf [Candidate]" term=3 +2024-07-05T13:37:26.039Z [ERROR] server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf: failed to make requestVote RPC: target="{Voter server-bb065309-aa4f-1089-caf8-d2abe062b8c3 bb065309-aa4f-1089-caf8-d2abe062b8c3}" error="failed to connect to peer: bb065309-aa4f-1089-caf8-d2abe062b8c3" term=3 +2024-07-05T13:37:26.039Z [ERROR] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: failed to appendEntries to: peer="{Voter server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf}" error="failed to connect to peer: 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf" +2024-07-05T13:37:26.039Z [ERROR] server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf: failed to make requestVote RPC: target="{Voter server-67170892-8a08-03c9-fd0c-1606d5dd00be 67170892-8a08-03c9-fd0c-1606d5dd00be}" error="failed to connect to peer: 67170892-8a08-03c9-fd0c-1606d5dd00be" term=3 +2024-07-05T13:37:26.057Z [ERROR] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: failed to heartbeat to: peer=8b34a5c7-035a-c25e-2c72-eaeccaffe1cf backoff time=25ms error="failed to connect to peer: 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf" +2024-07-05T13:37:26.062Z [WARN] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: failed to contact: server-id=server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf time=98.357245ms +2024-07-05T13:37:26.071Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: updating configuration: command=RemoveServer server-id=server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf server-addr= servers="[{Suffrage:Voter ID:server-67170892-8a08-03c9-fd0c-1606d5dd00be Address:67170892-8a08-03c9-fd0c-1606d5dd00be} {Suffrage:Voter ID:server-bb065309-aa4f-1089-caf8-d2abe062b8c3 Address:bb065309-aa4f-1089-caf8-d2abe062b8c3}]" +2024-07-05T13:37:26.072Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: removed peer, stopping replication: peer=server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf last-index=3 +2024-07-05T13:37:26.074Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: updating configuration: command=AddNonvoter server-id=server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf server-addr=8b34a5c7-035a-c25e-2c72-eaeccaffe1cf servers="[{Suffrage:Voter ID:server-67170892-8a08-03c9-fd0c-1606d5dd00be Address:67170892-8a08-03c9-fd0c-1606d5dd00be} {Suffrage:Voter ID:server-bb065309-aa4f-1089-caf8-d2abe062b8c3 Address:bb065309-aa4f-1089-caf8-d2abe062b8c3} {Suffrage:Nonvoter ID:server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf Address:8b34a5c7-035a-c25e-2c72-eaeccaffe1cf}]" +2024-07-05T13:37:26.075Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: added peer, starting replication: peer=server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf +2024-07-05T13:37:26.075Z [ERROR] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: failed to appendEntries to: peer="{Nonvoter server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf}" error="failed to connect to peer: 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf" +2024-07-05T13:37:26.085Z [ERROR] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: peer has newer term, stopping replication: peer="{Voter server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf}" +2024-07-05T13:37:26.085Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: entering follower state: follower="Node at bb065309-aa4f-1089-caf8-d2abe062b8c3 [Follower]" leader-address= leader-id= +2024-07-05T13:37:26.086Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: aborting pipeline replication: peer="{Voter server-67170892-8a08-03c9-fd0c-1606d5dd00be 67170892-8a08-03c9-fd0c-1606d5dd00be}" +2024-07-05T13:37:26.094Z [ERROR] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: peer has newer term, stopping replication: peer="{Nonvoter server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf}" +2024-07-05T13:37:26.106Z [WARN] server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf: Election timeout reached, restarting election +2024-07-05T13:37:26.107Z [INFO] server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf: entering candidate state: node="Node at 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf [Candidate]" term=23 +2024-07-05T13:37:26.107Z [WARN] server-67170892-8a08-03c9-fd0c-1606d5dd00be: rejecting vote request since we have a leader: from=8b34a5c7-035a-c25e-2c72-eaeccaffe1cf leader=bb065309-aa4f-1089-caf8-d2abe062b8c3 leader-id=server-bb065309-aa4f-1089-caf8-d2abe062b8c3 +2024-07-05T13:37:26.107Z [WARN] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: rejecting vote request since node is not a voter: from=8b34a5c7-035a-c25e-2c72-eaeccaffe1cf +2024-07-05T13:37:26.150Z [WARN] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:26.150Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: entering candidate state: node="Node at bb065309-aa4f-1089-caf8-d2abe062b8c3 [Candidate]" term=24 +2024-07-05T13:37:26.150Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: election won: term=24 tally=2 +2024-07-05T13:37:26.150Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: entering leader state: leader="Node at bb065309-aa4f-1089-caf8-d2abe062b8c3 [Leader]" +2024-07-05T13:37:26.150Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: added peer, starting replication: peer=server-67170892-8a08-03c9-fd0c-1606d5dd00be +2024-07-05T13:37:26.151Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: added peer, starting replication: peer=server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf +2024-07-05T13:37:26.151Z [WARN] server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf: failed to get previous log: previous-index=4 last-index=2 error="log not found" +2024-07-05T13:37:26.151Z [INFO] server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf: entering follower state: follower="Node at 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf [Follower]" leader-address=bb065309-aa4f-1089-caf8-d2abe062b8c3 leader-id=server-bb065309-aa4f-1089-caf8-d2abe062b8c3 +2024-07-05T13:37:26.151Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: pipelining replication: peer="{Voter server-67170892-8a08-03c9-fd0c-1606d5dd00be 67170892-8a08-03c9-fd0c-1606d5dd00be}" +2024-07-05T13:37:26.151Z [WARN] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: appendEntries rejected, sending older logs: peer="{Nonvoter server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf}" next=3 +2024-07-05T13:37:26.152Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: pipelining replication: peer="{Nonvoter server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf}" +2024-07-05T13:37:26.182Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:26.287Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:26.182500823 +0000 UTC m=+112.178421692 (1 nodes), 349.702µs from start of poll, 491.478234ms from cluster start. Timeout at 2024-07-05 13:37:26.287554905 +0000 UTC m=+112.283475773, 105.054081ms after stability +2024-07-05T13:37:26.384Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: updating configuration: command=RemoveServer server-id=server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf server-addr= servers="[{Suffrage:Voter ID:server-67170892-8a08-03c9-fd0c-1606d5dd00be Address:67170892-8a08-03c9-fd0c-1606d5dd00be} {Suffrage:Voter ID:server-bb065309-aa4f-1089-caf8-d2abe062b8c3 Address:bb065309-aa4f-1089-caf8-d2abe062b8c3}]" +2024-07-05T13:37:26.384Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: removed peer, stopping replication: peer=server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf last-index=106 +2024-07-05T13:37:26.384Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: aborting pipeline replication: peer="{Nonvoter server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf}" +2024-07-05T13:37:26.385Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: updating configuration: command=AddVoter server-id=server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf server-addr=8b34a5c7-035a-c25e-2c72-eaeccaffe1cf servers="[{Suffrage:Voter ID:server-67170892-8a08-03c9-fd0c-1606d5dd00be Address:67170892-8a08-03c9-fd0c-1606d5dd00be} {Suffrage:Voter ID:server-bb065309-aa4f-1089-caf8-d2abe062b8c3 Address:bb065309-aa4f-1089-caf8-d2abe062b8c3} {Suffrage:Voter ID:server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf Address:8b34a5c7-035a-c25e-2c72-eaeccaffe1cf}]" +2024-07-05T13:37:26.385Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: added peer, starting replication: peer=server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf +2024-07-05T13:37:26.386Z [WARN] server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf: failed to get previous log: previous-index=107 last-index=106 error="log not found" +2024-07-05T13:37:26.386Z [WARN] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: appendEntries rejected, sending older logs: peer="{Voter server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf}" next=107 +2024-07-05T13:37:26.386Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: pipelining replication: peer="{Voter server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf}" +2024-07-05T13:37:27.527Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:27.638Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:27.527868382 +0000 UTC m=+113.523788958 (1 nodes), 2.625µs from start of poll, 1.8368455s from cluster start. Timeout at 2024-07-05 13:37:27.633289665 +0000 UTC m=+113.629210825, 105.421867ms after stability +2024-07-05T13:37:27.639Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: aborting pipeline replication: peer="{Voter server-8b34a5c7-035a-c25e-2c72-eaeccaffe1cf 8b34a5c7-035a-c25e-2c72-eaeccaffe1cf}" +2024-07-05T13:37:27.639Z [INFO] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: aborting pipeline replication: peer="{Voter server-67170892-8a08-03c9-fd0c-1606d5dd00be 67170892-8a08-03c9-fd0c-1606d5dd00be}" +2024-07-05T13:37:28.140Z [ERROR] server-bb065309-aa4f-1089-caf8-d2abe062b8c3: failed to heartbeat to: peer=8b34a5c7-035a-c25e-2c72-eaeccaffe1cf backoff time=10ms error="command timed out" +--- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (2.46s) === RUN TestRaft_FollowerRemovalNoElection -2024-07-05T13:31:10.777Z [INFO] server-26394fb2-c496-bfd7-c3eb-dde1359d7923: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26394fb2-c496-bfd7-c3eb-dde1359d7923 Address:26394fb2-c496-bfd7-c3eb-dde1359d7923} {Suffrage:Voter ID:server-32333d4e-bb3d-408d-6ed0-3f5b70e2d5db Address:32333d4e-bb3d-408d-6ed0-3f5b70e2d5db} {Suffrage:Voter ID:server-3f278d0a-a6b5-1978-8745-aa30a4c61348 Address:3f278d0a-a6b5-1978-8745-aa30a4c61348}]" -2024-07-05T13:31:10.778Z [INFO] server-32333d4e-bb3d-408d-6ed0-3f5b70e2d5db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26394fb2-c496-bfd7-c3eb-dde1359d7923 Address:26394fb2-c496-bfd7-c3eb-dde1359d7923} {Suffrage:Voter ID:server-32333d4e-bb3d-408d-6ed0-3f5b70e2d5db Address:32333d4e-bb3d-408d-6ed0-3f5b70e2d5db} {Suffrage:Voter ID:server-3f278d0a-a6b5-1978-8745-aa30a4c61348 Address:3f278d0a-a6b5-1978-8745-aa30a4c61348}]" -2024-07-05T13:31:10.778Z [INFO] server-3f278d0a-a6b5-1978-8745-aa30a4c61348: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26394fb2-c496-bfd7-c3eb-dde1359d7923 Address:26394fb2-c496-bfd7-c3eb-dde1359d7923} {Suffrage:Voter ID:server-32333d4e-bb3d-408d-6ed0-3f5b70e2d5db Address:32333d4e-bb3d-408d-6ed0-3f5b70e2d5db} {Suffrage:Voter ID:server-3f278d0a-a6b5-1978-8745-aa30a4c61348 Address:3f278d0a-a6b5-1978-8745-aa30a4c61348}]" -2024-07-05T13:31:10.778Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:10.778Z [INFO] server-26394fb2-c496-bfd7-c3eb-dde1359d7923: entering follower state: follower="Node at 26394fb2-c496-bfd7-c3eb-dde1359d7923 [Follower]" leader-address= leader-id= -2024-07-05T13:31:10.778Z [INFO] server-32333d4e-bb3d-408d-6ed0-3f5b70e2d5db: entering follower state: follower="Node at 32333d4e-bb3d-408d-6ed0-3f5b70e2d5db [Follower]" leader-address= leader-id= -2024-07-05T13:31:10.778Z [INFO] server-3f278d0a-a6b5-1978-8745-aa30a4c61348: entering follower state: follower="Node at 3f278d0a-a6b5-1978-8745-aa30a4c61348 [Follower]" leader-address= leader-id= -2024-07-05T13:31:10.910Z [WARN] server-3f278d0a-a6b5-1978-8745-aa30a4c61348: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:10.910Z [INFO] server-3f278d0a-a6b5-1978-8745-aa30a4c61348: entering candidate state: node="Node at 3f278d0a-a6b5-1978-8745-aa30a4c61348 [Candidate]" term=2 -2024-07-05T13:31:10.910Z [INFO] server-3f278d0a-a6b5-1978-8745-aa30a4c61348: election won: term=2 tally=2 -2024-07-05T13:31:10.910Z [INFO] server-3f278d0a-a6b5-1978-8745-aa30a4c61348: entering leader state: leader="Node at 3f278d0a-a6b5-1978-8745-aa30a4c61348 [Leader]" -2024-07-05T13:31:10.910Z [INFO] server-3f278d0a-a6b5-1978-8745-aa30a4c61348: added peer, starting replication: peer=server-26394fb2-c496-bfd7-c3eb-dde1359d7923 -2024-07-05T13:31:10.910Z [INFO] server-3f278d0a-a6b5-1978-8745-aa30a4c61348: added peer, starting replication: peer=server-32333d4e-bb3d-408d-6ed0-3f5b70e2d5db -2024-07-05T13:31:10.911Z [INFO] server-3f278d0a-a6b5-1978-8745-aa30a4c61348: pipelining replication: peer="{Voter server-26394fb2-c496-bfd7-c3eb-dde1359d7923 26394fb2-c496-bfd7-c3eb-dde1359d7923}" -2024-07-05T13:31:10.911Z [INFO] server-3f278d0a-a6b5-1978-8745-aa30a4c61348: pipelining replication: peer="{Voter server-32333d4e-bb3d-408d-6ed0-3f5b70e2d5db 32333d4e-bb3d-408d-6ed0-3f5b70e2d5db}" -2024-07-05T13:31:11.116Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:10.910874275 +0000 UTC m=+111.680784950 (1 nodes), 132.567589ms from start of poll, 133.237085ms from cluster start. Timeout at 2024-07-05 13:31:11.116035757 +0000 UTC m=+111.885946432, 205.161482ms after stability -2024-07-05T13:31:11.116Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:11.321Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:11.11624147 +0000 UTC m=+111.886152145 (1 nodes), 2.24µs from start of poll, 338.60428ms from cluster start. Timeout at 2024-07-05 13:31:11.321593145 +0000 UTC m=+112.091503860, 205.351715ms after stability -2024-07-05T13:31:11.328Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:31:11.533Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:31:11.328768973 +0000 UTC m=+112.098679648 (2 nodes), 3µs from start of poll, 551.131783ms from cluster start. Timeout at 2024-07-05 13:31:11.53379926 +0000 UTC m=+112.303709935, 205.030287ms after stability -2024-07-05T13:31:11.533Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:11.739Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:11.533943775 +0000 UTC m=+112.303854450 (1 nodes), 1.48µs from start of poll, 756.306585ms from cluster start. Timeout at 2024-07-05 13:31:11.739033099 +0000 UTC m=+112.508943814, 205.089364ms after stability - raft_test.go:2961: [INFO] restarting Node at 26394fb2-c496-bfd7-c3eb-dde1359d7923 [Follower] -2024-07-05T13:31:11.739Z [INFO] server-26394fb2-c496-bfd7-c3eb-dde1359d7923: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26394fb2-c496-bfd7-c3eb-dde1359d7923 Address:26394fb2-c496-bfd7-c3eb-dde1359d7923} {Suffrage:Voter ID:server-32333d4e-bb3d-408d-6ed0-3f5b70e2d5db Address:32333d4e-bb3d-408d-6ed0-3f5b70e2d5db} {Suffrage:Voter ID:server-3f278d0a-a6b5-1978-8745-aa30a4c61348 Address:3f278d0a-a6b5-1978-8745-aa30a4c61348}]" -2024-07-05T13:31:11.739Z [INFO] server-26394fb2-c496-bfd7-c3eb-dde1359d7923: entering follower state: follower="Node at 26394fb2-c496-bfd7-c3eb-dde1359d7923 [Follower]" leader-address= leader-id= -2024-07-05T13:31:11.993Z [INFO] server-3f278d0a-a6b5-1978-8745-aa30a4c61348: aborting pipeline replication: peer="{Voter server-32333d4e-bb3d-408d-6ed0-3f5b70e2d5db 32333d4e-bb3d-408d-6ed0-3f5b70e2d5db}" -2024-07-05T13:31:12.376Z [ERROR] server-3f278d0a-a6b5-1978-8745-aa30a4c61348: failed to pipeline appendEntries: peer="{Voter server-26394fb2-c496-bfd7-c3eb-dde1359d7923 26394fb2-c496-bfd7-c3eb-dde1359d7923}" error="command enqueue timeout" -2024-07-05T13:31:12.376Z [INFO] server-3f278d0a-a6b5-1978-8745-aa30a4c61348: aborting pipeline replication: peer="{Voter server-26394fb2-c496-bfd7-c3eb-dde1359d7923 26394fb2-c496-bfd7-c3eb-dde1359d7923}" -2024-07-05T13:31:12.503Z [ERROR] server-3f278d0a-a6b5-1978-8745-aa30a4c61348: failed to heartbeat to: peer=26394fb2-c496-bfd7-c3eb-dde1359d7923 backoff time=10ms error="command timed out" ---- PASS: TestRaft_FollowerRemovalNoElection (1.73s) +2024-07-05T13:37:28.149Z [INFO] server-e697c4b6-7329-6a4d-e164-a8d5f111e2a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e697c4b6-7329-6a4d-e164-a8d5f111e2a8 Address:e697c4b6-7329-6a4d-e164-a8d5f111e2a8} {Suffrage:Voter ID:server-57eb3391-d90f-caa4-5cd8-a03574347197 Address:57eb3391-d90f-caa4-5cd8-a03574347197} {Suffrage:Voter ID:server-3e068666-d104-d4c0-47b3-5d6e7c5ce543 Address:3e068666-d104-d4c0-47b3-5d6e7c5ce543}]" +2024-07-05T13:37:28.150Z [INFO] server-e697c4b6-7329-6a4d-e164-a8d5f111e2a8: entering follower state: follower="Node at e697c4b6-7329-6a4d-e164-a8d5f111e2a8 [Follower]" leader-address= leader-id= +2024-07-05T13:37:28.150Z [INFO] server-57eb3391-d90f-caa4-5cd8-a03574347197: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e697c4b6-7329-6a4d-e164-a8d5f111e2a8 Address:e697c4b6-7329-6a4d-e164-a8d5f111e2a8} {Suffrage:Voter ID:server-57eb3391-d90f-caa4-5cd8-a03574347197 Address:57eb3391-d90f-caa4-5cd8-a03574347197} {Suffrage:Voter ID:server-3e068666-d104-d4c0-47b3-5d6e7c5ce543 Address:3e068666-d104-d4c0-47b3-5d6e7c5ce543}]" +2024-07-05T13:37:28.150Z [INFO] server-57eb3391-d90f-caa4-5cd8-a03574347197: entering follower state: follower="Node at 57eb3391-d90f-caa4-5cd8-a03574347197 [Follower]" leader-address= leader-id= +2024-07-05T13:37:28.151Z [INFO] server-3e068666-d104-d4c0-47b3-5d6e7c5ce543: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e697c4b6-7329-6a4d-e164-a8d5f111e2a8 Address:e697c4b6-7329-6a4d-e164-a8d5f111e2a8} {Suffrage:Voter ID:server-57eb3391-d90f-caa4-5cd8-a03574347197 Address:57eb3391-d90f-caa4-5cd8-a03574347197} {Suffrage:Voter ID:server-3e068666-d104-d4c0-47b3-5d6e7c5ce543 Address:3e068666-d104-d4c0-47b3-5d6e7c5ce543}]" +2024-07-05T13:37:28.151Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:28.151Z [INFO] server-3e068666-d104-d4c0-47b3-5d6e7c5ce543: entering follower state: follower="Node at 3e068666-d104-d4c0-47b3-5d6e7c5ce543 [Follower]" leader-address= leader-id= +2024-07-05T13:37:28.257Z [WARN] server-e697c4b6-7329-6a4d-e164-a8d5f111e2a8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:28.257Z [INFO] server-e697c4b6-7329-6a4d-e164-a8d5f111e2a8: entering candidate state: node="Node at e697c4b6-7329-6a4d-e164-a8d5f111e2a8 [Candidate]" term=2 +2024-07-05T13:37:28.257Z [INFO] server-e697c4b6-7329-6a4d-e164-a8d5f111e2a8: election won: term=2 tally=2 +2024-07-05T13:37:28.257Z [INFO] server-e697c4b6-7329-6a4d-e164-a8d5f111e2a8: entering leader state: leader="Node at e697c4b6-7329-6a4d-e164-a8d5f111e2a8 [Leader]" +2024-07-05T13:37:28.258Z [INFO] server-e697c4b6-7329-6a4d-e164-a8d5f111e2a8: added peer, starting replication: peer=server-57eb3391-d90f-caa4-5cd8-a03574347197 +2024-07-05T13:37:28.258Z [INFO] server-e697c4b6-7329-6a4d-e164-a8d5f111e2a8: added peer, starting replication: peer=server-3e068666-d104-d4c0-47b3-5d6e7c5ce543 +2024-07-05T13:37:28.258Z [INFO] server-e697c4b6-7329-6a4d-e164-a8d5f111e2a8: pipelining replication: peer="{Voter server-57eb3391-d90f-caa4-5cd8-a03574347197 57eb3391-d90f-caa4-5cd8-a03574347197}" +2024-07-05T13:37:28.258Z [INFO] server-e697c4b6-7329-6a4d-e164-a8d5f111e2a8: pipelining replication: peer="{Voter server-3e068666-d104-d4c0-47b3-5d6e7c5ce543 3e068666-d104-d4c0-47b3-5d6e7c5ce543}" +2024-07-05T13:37:28.463Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:28.2579778 +0000 UTC m=+114.253898960 (1 nodes), 106.737551ms from start of poll, 108.708307ms from cluster start. Timeout at 2024-07-05 13:37:28.46299064 +0000 UTC m=+114.458911800, 205.01284ms after stability +2024-07-05T13:37:28.463Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:28.669Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:28.46381575 +0000 UTC m=+114.459737201 (1 nodes), 6.125µs from start of poll, 314.546548ms from cluster start. Timeout at 2024-07-05 13:37:28.668874963 +0000 UTC m=+114.664796415, 205.059214ms after stability +2024-07-05T13:37:28.674Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:28.879Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:28.674557985 +0000 UTC m=+114.670479436 (2 nodes), 8.458µs from start of poll, 525.288783ms from cluster start. Timeout at 2024-07-05 13:37:28.879590366 +0000 UTC m=+114.875511234, 205.031798ms after stability +2024-07-05T13:37:28.879Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:29.085Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:28.879886985 +0000 UTC m=+114.875808145 (1 nodes), 5.25µs from start of poll, 730.617492ms from cluster start. Timeout at 2024-07-05 13:37:29.085467398 +0000 UTC m=+115.081388266, 205.580121ms after stability + raft_test.go:2961: [INFO] restarting Node at 57eb3391-d90f-caa4-5cd8-a03574347197 [Follower] +2024-07-05T13:37:29.086Z [INFO] server-57eb3391-d90f-caa4-5cd8-a03574347197: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e697c4b6-7329-6a4d-e164-a8d5f111e2a8 Address:e697c4b6-7329-6a4d-e164-a8d5f111e2a8} {Suffrage:Voter ID:server-57eb3391-d90f-caa4-5cd8-a03574347197 Address:57eb3391-d90f-caa4-5cd8-a03574347197} {Suffrage:Voter ID:server-3e068666-d104-d4c0-47b3-5d6e7c5ce543 Address:3e068666-d104-d4c0-47b3-5d6e7c5ce543}]" +2024-07-05T13:37:29.086Z [INFO] server-57eb3391-d90f-caa4-5cd8-a03574347197: entering follower state: follower="Node at 57eb3391-d90f-caa4-5cd8-a03574347197 [Follower]" leader-address= leader-id= +2024-07-05T13:37:29.337Z [INFO] server-e697c4b6-7329-6a4d-e164-a8d5f111e2a8: aborting pipeline replication: peer="{Voter server-3e068666-d104-d4c0-47b3-5d6e7c5ce543 3e068666-d104-d4c0-47b3-5d6e7c5ce543}" +2024-07-05T13:37:29.741Z [ERROR] server-e697c4b6-7329-6a4d-e164-a8d5f111e2a8: failed to pipeline appendEntries: peer="{Voter server-57eb3391-d90f-caa4-5cd8-a03574347197 57eb3391-d90f-caa4-5cd8-a03574347197}" error="command enqueue timeout" +2024-07-05T13:37:29.741Z [INFO] server-e697c4b6-7329-6a4d-e164-a8d5f111e2a8: aborting pipeline replication: peer="{Voter server-57eb3391-d90f-caa4-5cd8-a03574347197 57eb3391-d90f-caa4-5cd8-a03574347197}" +2024-07-05T13:37:29.855Z [ERROR] server-e697c4b6-7329-6a4d-e164-a8d5f111e2a8: failed to heartbeat to: peer=57eb3391-d90f-caa4-5cd8-a03574347197 backoff time=10ms error="command timed out" +--- PASS: TestRaft_FollowerRemovalNoElection (1.72s) === RUN TestRaft_VoteWithNoIDNoAddr -2024-07-05T13:31:12.508Z [INFO] server-5ad00174-f857-96c1-a68d-565e23366667: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ad00174-f857-96c1-a68d-565e23366667 Address:5ad00174-f857-96c1-a68d-565e23366667} {Suffrage:Voter ID:server-fa577132-3cc0-a25f-7329-f0e87cb90f78 Address:fa577132-3cc0-a25f-7329-f0e87cb90f78} {Suffrage:Voter ID:server-d214fbbd-75bf-b273-e878-ebe93491867a Address:d214fbbd-75bf-b273-e878-ebe93491867a}]" -2024-07-05T13:31:12.508Z [INFO] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ad00174-f857-96c1-a68d-565e23366667 Address:5ad00174-f857-96c1-a68d-565e23366667} {Suffrage:Voter ID:server-fa577132-3cc0-a25f-7329-f0e87cb90f78 Address:fa577132-3cc0-a25f-7329-f0e87cb90f78} {Suffrage:Voter ID:server-d214fbbd-75bf-b273-e878-ebe93491867a Address:d214fbbd-75bf-b273-e878-ebe93491867a}]" -2024-07-05T13:31:12.509Z [INFO] server-d214fbbd-75bf-b273-e878-ebe93491867a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ad00174-f857-96c1-a68d-565e23366667 Address:5ad00174-f857-96c1-a68d-565e23366667} {Suffrage:Voter ID:server-fa577132-3cc0-a25f-7329-f0e87cb90f78 Address:fa577132-3cc0-a25f-7329-f0e87cb90f78} {Suffrage:Voter ID:server-d214fbbd-75bf-b273-e878-ebe93491867a Address:d214fbbd-75bf-b273-e878-ebe93491867a}]" -2024-07-05T13:31:12.509Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:12.509Z [INFO] server-5ad00174-f857-96c1-a68d-565e23366667: entering follower state: follower="Node at 5ad00174-f857-96c1-a68d-565e23366667 [Follower]" leader-address= leader-id= -2024-07-05T13:31:12.509Z [INFO] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: entering follower state: follower="Node at fa577132-3cc0-a25f-7329-f0e87cb90f78 [Follower]" leader-address= leader-id= -2024-07-05T13:31:12.509Z [INFO] server-d214fbbd-75bf-b273-e878-ebe93491867a: entering follower state: follower="Node at d214fbbd-75bf-b273-e878-ebe93491867a [Follower]" leader-address= leader-id= -2024-07-05T13:31:12.564Z [WARN] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2024-07-05T13:31:12.564Z [INFO] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: entering candidate state: node="Node at fa577132-3cc0-a25f-7329-f0e87cb90f78 [Candidate]" term=2 -2024-07-05T13:31:12.564Z [INFO] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: election won: term=2 tally=2 -2024-07-05T13:31:12.564Z [INFO] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: entering leader state: leader="Node at fa577132-3cc0-a25f-7329-f0e87cb90f78 [Leader]" -2024-07-05T13:31:12.564Z [INFO] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: added peer, starting replication: peer=server-5ad00174-f857-96c1-a68d-565e23366667 -2024-07-05T13:31:12.564Z [INFO] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: added peer, starting replication: peer=server-d214fbbd-75bf-b273-e878-ebe93491867a -2024-07-05T13:31:12.564Z [INFO] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: pipelining replication: peer="{Voter server-5ad00174-f857-96c1-a68d-565e23366667 5ad00174-f857-96c1-a68d-565e23366667}" -2024-07-05T13:31:12.564Z [INFO] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: pipelining replication: peer="{Voter server-d214fbbd-75bf-b273-e878-ebe93491867a d214fbbd-75bf-b273-e878-ebe93491867a}" -2024-07-05T13:31:12.669Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:12.564459859 +0000 UTC m=+113.334370534 (1 nodes), 55.360858ms from start of poll, 56.088792ms from cluster start. Timeout at 2024-07-05 13:31:12.669710526 +0000 UTC m=+113.439621201, 105.250667ms after stability -2024-07-05T13:31:12.669Z [INFO] cluster: starting stability test: raft-state=Leader -2024-07-05T13:31:12.775Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:31:12.669873801 +0000 UTC m=+113.439784476 (1 nodes), 2.12µs from start of poll, 161.502734ms from cluster start. Timeout at 2024-07-05 13:31:12.775047951 +0000 UTC m=+113.544958626, 105.17415ms after stability -2024-07-05T13:31:12.781Z [INFO] cluster: starting stability test: raft-state=Follower -2024-07-05T13:31:12.886Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:31:12.781263852 +0000 UTC m=+113.551174527 (2 nodes), 2.119µs from start of poll, 272.892785ms from cluster start. Timeout at 2024-07-05 13:31:12.886540519 +0000 UTC m=+113.656451194, 105.276667ms after stability -2024-07-05T13:31:12.886Z [ERROR] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: failed to heartbeat to: peer=5ad00174-f857-96c1-a68d-565e23366667 backoff time=10ms error="failed to connect to peer: 5ad00174-f857-96c1-a68d-565e23366667" -2024-07-05T13:31:12.888Z [ERROR] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: failed to pipeline appendEntries: peer="{Voter server-5ad00174-f857-96c1-a68d-565e23366667 5ad00174-f857-96c1-a68d-565e23366667}" error="append pipeline closed" -2024-07-05T13:31:12.888Z [INFO] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: aborting pipeline replication: peer="{Voter server-5ad00174-f857-96c1-a68d-565e23366667 5ad00174-f857-96c1-a68d-565e23366667}" -2024-07-05T13:31:12.888Z [ERROR] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: failed to heartbeat to: peer=d214fbbd-75bf-b273-e878-ebe93491867a backoff time=10ms error="failed to connect to peer: d214fbbd-75bf-b273-e878-ebe93491867a" -2024-07-05T13:31:12.893Z [ERROR] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: failed to pipeline appendEntries: peer="{Voter server-d214fbbd-75bf-b273-e878-ebe93491867a d214fbbd-75bf-b273-e878-ebe93491867a}" error="append pipeline closed" -2024-07-05T13:31:12.893Z [INFO] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: aborting pipeline replication: peer="{Voter server-d214fbbd-75bf-b273-e878-ebe93491867a d214fbbd-75bf-b273-e878-ebe93491867a}" -2024-07-05T13:31:12.896Z [ERROR] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: failed to appendEntries to: peer="{Voter server-5ad00174-f857-96c1-a68d-565e23366667 5ad00174-f857-96c1-a68d-565e23366667}" error="failed to connect to peer: 5ad00174-f857-96c1-a68d-565e23366667" -2024-07-05T13:31:12.902Z [ERROR] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: failed to appendEntries to: peer="{Voter server-d214fbbd-75bf-b273-e878-ebe93491867a d214fbbd-75bf-b273-e878-ebe93491867a}" error="failed to connect to peer: d214fbbd-75bf-b273-e878-ebe93491867a" -2024-07-05T13:31:12.905Z [ERROR] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: failed to heartbeat to: peer=d214fbbd-75bf-b273-e878-ebe93491867a backoff time=10ms error="failed to connect to peer: d214fbbd-75bf-b273-e878-ebe93491867a" -2024-07-05T13:31:12.905Z [ERROR] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: failed to heartbeat to: peer=5ad00174-f857-96c1-a68d-565e23366667 backoff time=10ms error="failed to connect to peer: 5ad00174-f857-96c1-a68d-565e23366667" -2024-07-05T13:31:12.921Z [ERROR] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: failed to appendEntries to: peer="{Voter server-d214fbbd-75bf-b273-e878-ebe93491867a d214fbbd-75bf-b273-e878-ebe93491867a}" error="failed to connect to peer: d214fbbd-75bf-b273-e878-ebe93491867a" -2024-07-05T13:31:12.929Z [ERROR] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: failed to appendEntries to: peer="{Voter server-5ad00174-f857-96c1-a68d-565e23366667 5ad00174-f857-96c1-a68d-565e23366667}" error="failed to connect to peer: 5ad00174-f857-96c1-a68d-565e23366667" -2024-07-05T13:31:12.931Z [WARN] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: failed to contact: server-id=server-5ad00174-f857-96c1-a68d-565e23366667 time=50.035684ms -2024-07-05T13:31:12.932Z [ERROR] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: failed to heartbeat to: peer=d214fbbd-75bf-b273-e878-ebe93491867a backoff time=10ms error="failed to connect to peer: d214fbbd-75bf-b273-e878-ebe93491867a" -2024-07-05T13:31:12.932Z [ERROR] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: failed to heartbeat to: peer=5ad00174-f857-96c1-a68d-565e23366667 backoff time=10ms error="failed to connect to peer: 5ad00174-f857-96c1-a68d-565e23366667" -2024-07-05T13:31:12.940Z [ERROR] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: failed to appendEntries to: peer="{Voter server-d214fbbd-75bf-b273-e878-ebe93491867a d214fbbd-75bf-b273-e878-ebe93491867a}" error="failed to connect to peer: d214fbbd-75bf-b273-e878-ebe93491867a" -2024-07-05T13:31:12.941Z [WARN] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: failed to contact: server-id=server-5ad00174-f857-96c1-a68d-565e23366667 time=60.248646ms -2024-07-05T13:31:12.941Z [WARN] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: failed to contact: server-id=server-d214fbbd-75bf-b273-e878-ebe93491867a time=57.146315ms -2024-07-05T13:31:12.941Z [WARN] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: failed to contact quorum of nodes, stepping down -2024-07-05T13:31:12.941Z [INFO] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: entering follower state: follower="Node at fa577132-3cc0-a25f-7329-f0e87cb90f78 [Follower]" leader-address= leader-id= -2024-07-05T13:31:12.948Z [ERROR] server-fa577132-3cc0-a25f-7329-f0e87cb90f78: failed to appendEntries to: peer="{Voter server-5ad00174-f857-96c1-a68d-565e23366667 5ad00174-f857-96c1-a68d-565e23366667}" error="failed to connect to peer: 5ad00174-f857-96c1-a68d-565e23366667" -2024-07-05T13:31:12.958Z [WARN] server-5ad00174-f857-96c1-a68d-565e23366667: heartbeat timeout reached, starting election: last-leader-addr=fa577132-3cc0-a25f-7329-f0e87cb90f78 last-leader-id=server-fa577132-3cc0-a25f-7329-f0e87cb90f78 -2024-07-05T13:31:12.958Z [INFO] server-5ad00174-f857-96c1-a68d-565e23366667: entering candidate state: node="Node at 5ad00174-f857-96c1-a68d-565e23366667 [Candidate]" term=3 -2024-07-05T13:31:12.958Z [WARN] server-d214fbbd-75bf-b273-e878-ebe93491867a: rejecting vote request since we have a leader: from=5ad00174-f857-96c1-a68d-565e23366667 leader=fa577132-3cc0-a25f-7329-f0e87cb90f78 leader-id=server-fa577132-3cc0-a25f-7329-f0e87cb90f78 -2024-07-05T13:31:12.958Z [ERROR] server-5ad00174-f857-96c1-a68d-565e23366667: failed to make requestVote RPC: target="{Voter server-fa577132-3cc0-a25f-7329-f0e87cb90f78 fa577132-3cc0-a25f-7329-f0e87cb90f78}" error="failed to connect to peer: fa577132-3cc0-a25f-7329-f0e87cb90f78" term=3 -2024-07-05T13:31:12.992Z [INFO] server-5ad00174-f857-96c1-a68d-565e23366667: entering follower state: follower="Node at 5ad00174-f857-96c1-a68d-565e23366667 [Follower]" leader-address= leader-id= ---- PASS: TestRaft_VoteWithNoIDNoAddr (0.49s) +2024-07-05T13:37:29.870Z [INFO] server-9d01a0ef-e419-5fab-5d55-30cfa406989c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d01a0ef-e419-5fab-5d55-30cfa406989c Address:9d01a0ef-e419-5fab-5d55-30cfa406989c} {Suffrage:Voter ID:server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c Address:e7b45ff5-d5ba-158b-de42-1d8f804cdf0c} {Suffrage:Voter ID:server-bca3d162-8f10-9e57-8704-4ceaa94cce66 Address:bca3d162-8f10-9e57-8704-4ceaa94cce66}]" +2024-07-05T13:37:29.870Z [INFO] server-9d01a0ef-e419-5fab-5d55-30cfa406989c: entering follower state: follower="Node at 9d01a0ef-e419-5fab-5d55-30cfa406989c [Follower]" leader-address= leader-id= +2024-07-05T13:37:29.870Z [INFO] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d01a0ef-e419-5fab-5d55-30cfa406989c Address:9d01a0ef-e419-5fab-5d55-30cfa406989c} {Suffrage:Voter ID:server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c Address:e7b45ff5-d5ba-158b-de42-1d8f804cdf0c} {Suffrage:Voter ID:server-bca3d162-8f10-9e57-8704-4ceaa94cce66 Address:bca3d162-8f10-9e57-8704-4ceaa94cce66}]" +2024-07-05T13:37:29.871Z [INFO] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: entering follower state: follower="Node at e7b45ff5-d5ba-158b-de42-1d8f804cdf0c [Follower]" leader-address= leader-id= +2024-07-05T13:37:29.871Z [INFO] server-bca3d162-8f10-9e57-8704-4ceaa94cce66: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d01a0ef-e419-5fab-5d55-30cfa406989c Address:9d01a0ef-e419-5fab-5d55-30cfa406989c} {Suffrage:Voter ID:server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c Address:e7b45ff5-d5ba-158b-de42-1d8f804cdf0c} {Suffrage:Voter ID:server-bca3d162-8f10-9e57-8704-4ceaa94cce66 Address:bca3d162-8f10-9e57-8704-4ceaa94cce66}]" +2024-07-05T13:37:29.871Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:29.871Z [INFO] server-bca3d162-8f10-9e57-8704-4ceaa94cce66: entering follower state: follower="Node at bca3d162-8f10-9e57-8704-4ceaa94cce66 [Follower]" leader-address= leader-id= +2024-07-05T13:37:29.934Z [WARN] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:29.934Z [INFO] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: entering candidate state: node="Node at e7b45ff5-d5ba-158b-de42-1d8f804cdf0c [Candidate]" term=2 +2024-07-05T13:37:29.938Z [INFO] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: election won: term=2 tally=2 +2024-07-05T13:37:29.938Z [INFO] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: entering leader state: leader="Node at e7b45ff5-d5ba-158b-de42-1d8f804cdf0c [Leader]" +2024-07-05T13:37:29.939Z [INFO] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: added peer, starting replication: peer=server-9d01a0ef-e419-5fab-5d55-30cfa406989c +2024-07-05T13:37:29.939Z [INFO] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: added peer, starting replication: peer=server-bca3d162-8f10-9e57-8704-4ceaa94cce66 +2024-07-05T13:37:29.939Z [INFO] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: pipelining replication: peer="{Voter server-bca3d162-8f10-9e57-8704-4ceaa94cce66 bca3d162-8f10-9e57-8704-4ceaa94cce66}" +2024-07-05T13:37:29.940Z [INFO] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: pipelining replication: peer="{Voter server-9d01a0ef-e419-5fab-5d55-30cfa406989c 9d01a0ef-e419-5fab-5d55-30cfa406989c}" +2024-07-05T13:37:30.045Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:29.93935439 +0000 UTC m=+115.935275549 (1 nodes), 67.656389ms from start of poll, 69.614895ms from cluster start. Timeout at 2024-07-05 13:37:30.045131791 +0000 UTC m=+116.041053243, 105.777694ms after stability +2024-07-05T13:37:30.045Z [INFO] cluster: starting stability test: raft-state=Leader +2024-07-05T13:37:30.151Z [INFO] cluster: stable state for Leader reached at 2024-07-05 13:37:30.045651824 +0000 UTC m=+116.041573275 (1 nodes), 6.708µs from start of poll, 175.912621ms from cluster start. Timeout at 2024-07-05 13:37:30.151207563 +0000 UTC m=+116.147128723, 105.555448ms after stability +2024-07-05T13:37:30.156Z [INFO] cluster: starting stability test: raft-state=Follower +2024-07-05T13:37:30.261Z [INFO] cluster: stable state for Follower reached at 2024-07-05 13:37:30.156615547 +0000 UTC m=+116.152536416 (2 nodes), 4.959µs from start of poll, 286.875762ms from cluster start. Timeout at 2024-07-05 13:37:30.261675754 +0000 UTC m=+116.257597206, 105.06079ms after stability +2024-07-05T13:37:30.262Z [ERROR] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: failed to pipeline appendEntries: peer="{Voter server-9d01a0ef-e419-5fab-5d55-30cfa406989c 9d01a0ef-e419-5fab-5d55-30cfa406989c}" error="append pipeline closed" +2024-07-05T13:37:30.262Z [INFO] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: aborting pipeline replication: peer="{Voter server-9d01a0ef-e419-5fab-5d55-30cfa406989c 9d01a0ef-e419-5fab-5d55-30cfa406989c}" +2024-07-05T13:37:30.269Z [ERROR] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: failed to appendEntries to: peer="{Voter server-9d01a0ef-e419-5fab-5d55-30cfa406989c 9d01a0ef-e419-5fab-5d55-30cfa406989c}" error="failed to connect to peer: 9d01a0ef-e419-5fab-5d55-30cfa406989c" +2024-07-05T13:37:30.277Z [ERROR] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: failed to heartbeat to: peer=9d01a0ef-e419-5fab-5d55-30cfa406989c backoff time=10ms error="failed to connect to peer: 9d01a0ef-e419-5fab-5d55-30cfa406989c" +2024-07-05T13:37:30.278Z [ERROR] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: failed to pipeline appendEntries: peer="{Voter server-bca3d162-8f10-9e57-8704-4ceaa94cce66 bca3d162-8f10-9e57-8704-4ceaa94cce66}" error="append pipeline closed" +2024-07-05T13:37:30.279Z [INFO] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: aborting pipeline replication: peer="{Voter server-bca3d162-8f10-9e57-8704-4ceaa94cce66 bca3d162-8f10-9e57-8704-4ceaa94cce66}" +2024-07-05T13:37:30.279Z [ERROR] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: failed to heartbeat to: peer=bca3d162-8f10-9e57-8704-4ceaa94cce66 backoff time=10ms error="failed to connect to peer: bca3d162-8f10-9e57-8704-4ceaa94cce66" +2024-07-05T13:37:30.287Z [ERROR] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: failed to appendEntries to: peer="{Voter server-bca3d162-8f10-9e57-8704-4ceaa94cce66 bca3d162-8f10-9e57-8704-4ceaa94cce66}" error="failed to connect to peer: bca3d162-8f10-9e57-8704-4ceaa94cce66" +2024-07-05T13:37:30.289Z [ERROR] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: failed to appendEntries to: peer="{Voter server-9d01a0ef-e419-5fab-5d55-30cfa406989c 9d01a0ef-e419-5fab-5d55-30cfa406989c}" error="failed to connect to peer: 9d01a0ef-e419-5fab-5d55-30cfa406989c" +2024-07-05T13:37:30.296Z [ERROR] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: failed to heartbeat to: peer=9d01a0ef-e419-5fab-5d55-30cfa406989c backoff time=10ms error="failed to connect to peer: 9d01a0ef-e419-5fab-5d55-30cfa406989c" +2024-07-05T13:37:30.299Z [ERROR] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: failed to heartbeat to: peer=bca3d162-8f10-9e57-8704-4ceaa94cce66 backoff time=10ms error="failed to connect to peer: bca3d162-8f10-9e57-8704-4ceaa94cce66" +2024-07-05T13:37:30.306Z [ERROR] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: failed to appendEntries to: peer="{Voter server-bca3d162-8f10-9e57-8704-4ceaa94cce66 bca3d162-8f10-9e57-8704-4ceaa94cce66}" error="failed to connect to peer: bca3d162-8f10-9e57-8704-4ceaa94cce66" +2024-07-05T13:37:30.307Z [ERROR] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: failed to appendEntries to: peer="{Voter server-9d01a0ef-e419-5fab-5d55-30cfa406989c 9d01a0ef-e419-5fab-5d55-30cfa406989c}" error="failed to connect to peer: 9d01a0ef-e419-5fab-5d55-30cfa406989c" +2024-07-05T13:37:30.311Z [WARN] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: failed to contact: server-id=server-bca3d162-8f10-9e57-8704-4ceaa94cce66 time=50.172459ms +2024-07-05T13:37:30.316Z [ERROR] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: failed to heartbeat to: peer=9d01a0ef-e419-5fab-5d55-30cfa406989c backoff time=10ms error="failed to connect to peer: 9d01a0ef-e419-5fab-5d55-30cfa406989c" +2024-07-05T13:37:30.319Z [ERROR] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: failed to heartbeat to: peer=bca3d162-8f10-9e57-8704-4ceaa94cce66 backoff time=10ms error="failed to connect to peer: bca3d162-8f10-9e57-8704-4ceaa94cce66" +2024-07-05T13:37:30.321Z [WARN] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: failed to contact: server-id=server-9d01a0ef-e419-5fab-5d55-30cfa406989c time=60.159234ms +2024-07-05T13:37:30.321Z [WARN] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: failed to contact: server-id=server-bca3d162-8f10-9e57-8704-4ceaa94cce66 time=60.800889ms +2024-07-05T13:37:30.321Z [WARN] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: failed to contact quorum of nodes, stepping down +2024-07-05T13:37:30.322Z [INFO] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: entering follower state: follower="Node at e7b45ff5-d5ba-158b-de42-1d8f804cdf0c [Follower]" leader-address= leader-id= +2024-07-05T13:37:30.325Z [ERROR] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: failed to appendEntries to: peer="{Voter server-bca3d162-8f10-9e57-8704-4ceaa94cce66 bca3d162-8f10-9e57-8704-4ceaa94cce66}" error="failed to connect to peer: bca3d162-8f10-9e57-8704-4ceaa94cce66" +2024-07-05T13:37:30.335Z [ERROR] server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c: failed to appendEntries to: peer="{Voter server-9d01a0ef-e419-5fab-5d55-30cfa406989c 9d01a0ef-e419-5fab-5d55-30cfa406989c}" error="failed to connect to peer: 9d01a0ef-e419-5fab-5d55-30cfa406989c" +2024-07-05T13:37:30.366Z [WARN] server-9d01a0ef-e419-5fab-5d55-30cfa406989c: heartbeat timeout reached, starting election: last-leader-addr=e7b45ff5-d5ba-158b-de42-1d8f804cdf0c last-leader-id=server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c +2024-07-05T13:37:30.366Z [INFO] server-9d01a0ef-e419-5fab-5d55-30cfa406989c: entering candidate state: node="Node at 9d01a0ef-e419-5fab-5d55-30cfa406989c [Candidate]" term=3 +2024-07-05T13:37:30.366Z [ERROR] server-9d01a0ef-e419-5fab-5d55-30cfa406989c: failed to make requestVote RPC: target="{Voter server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c e7b45ff5-d5ba-158b-de42-1d8f804cdf0c}" error="failed to connect to peer: e7b45ff5-d5ba-158b-de42-1d8f804cdf0c" term=3 +2024-07-05T13:37:30.366Z [WARN] server-bca3d162-8f10-9e57-8704-4ceaa94cce66: rejecting vote request since we have a leader: from=9d01a0ef-e419-5fab-5d55-30cfa406989c leader=e7b45ff5-d5ba-158b-de42-1d8f804cdf0c leader-id=server-e7b45ff5-d5ba-158b-de42-1d8f804cdf0c +2024-07-05T13:37:30.367Z [INFO] server-9d01a0ef-e419-5fab-5d55-30cfa406989c: entering follower state: follower="Node at 9d01a0ef-e419-5fab-5d55-30cfa406989c [Follower]" leader-address= leader-id= +--- PASS: TestRaft_VoteWithNoIDNoAddr (0.50s) === RUN TestRaft_runFollower_State_Transition === RUN TestRaft_runFollower_State_Transition/NonVoter -2024-07-05T13:31:12.994Z [INFO] 127.0.0.1:46463 :: starting node: addr=127.0.0.1:46463 -2024-07-05T13:31:12.994Z [INFO] 127.0.0.1:46463 :: initial configuration: index=0 servers=[] -2024-07-05T13:31:12.994Z [INFO] 127.0.0.1:46463 :: entering follower state: follower="Node at 127.0.0.1:46463 [Follower]" leader-address= leader-id= -2024-07-05T13:31:13.051Z [WARN] 127.0.0.1:46463 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election +2024-07-05T13:37:30.373Z [INFO] 127.0.0.1:44443 :: starting node: addr=127.0.0.1:44443 +2024-07-05T13:37:30.373Z [INFO] 127.0.0.1:44443 :: initial configuration: index=0 servers=[] +2024-07-05T13:37:30.373Z [INFO] 127.0.0.1:44443 :: entering follower state: follower="Node at 127.0.0.1:44443 [Follower]" leader-address= leader-id= +2024-07-05T13:37:30.476Z [WARN] 127.0.0.1:44443 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election === RUN TestRaft_runFollower_State_Transition/Voter -2024-07-05T13:31:13.155Z [INFO] 127.0.0.1:40807 :: starting node: addr=127.0.0.1:40807 -2024-07-05T13:31:13.155Z [INFO] 127.0.0.1:40807 :: initial configuration: index=0 servers=[] -2024-07-05T13:31:13.155Z [INFO] 127.0.0.1:40807 :: entering follower state: follower="Node at 127.0.0.1:40807 [Follower]" leader-address= leader-id= -2024-07-05T13:31:13.240Z [WARN] 127.0.0.1:40807 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2024-07-05T13:37:30.525Z [INFO] 127.0.0.1:39761 :: starting node: addr=127.0.0.1:39761 +2024-07-05T13:37:30.525Z [INFO] 127.0.0.1:39761 :: initial configuration: index=0 servers=[] +2024-07-05T13:37:30.525Z [INFO] 127.0.0.1:39761 :: entering follower state: follower="Node at 127.0.0.1:39761 [Follower]" leader-address= leader-id= +2024-07-05T13:37:30.615Z [WARN] 127.0.0.1:39761 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= === RUN TestRaft_runFollower_State_Transition/Not_in_Config -2024-07-05T13:31:13.307Z [INFO] 127.0.0.1:35477 :: starting node: addr=127.0.0.1:35477 -2024-07-05T13:31:13.307Z [INFO] 127.0.0.1:35477 :: initial configuration: index=0 servers=[] -2024-07-05T13:31:13.307Z [INFO] 127.0.0.1:35477 :: entering follower state: follower="Node at 127.0.0.1:35477 [Follower]" leader-address= leader-id= -2024-07-05T13:31:13.404Z [WARN] 127.0.0.1:35477 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election +2024-07-05T13:37:30.677Z [INFO] 127.0.0.1:43255 :: starting node: addr=127.0.0.1:43255 +2024-07-05T13:37:30.678Z [INFO] 127.0.0.1:43255 :: initial configuration: index=0 servers=[] +2024-07-05T13:37:30.678Z [INFO] 127.0.0.1:43255 :: entering follower state: follower="Node at 127.0.0.1:43255 [Follower]" leader-address= leader-id= +2024-07-05T13:37:30.762Z [WARN] 127.0.0.1:43255 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election --- PASS: TestRaft_runFollower_State_Transition (0.46s) - --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.16s) + --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s) --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s) --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s) === RUN TestRaft_runFollower_ReloadTimeoutConfigs -2024-07-05T13:31:13.458Z [INFO] 127.0.0.1:37833 :: starting node: addr=127.0.0.1:37833 -2024-07-05T13:31:13.458Z [INFO] 127.0.0.1:37833 :: initial configuration: index=0 servers=[] -2024-07-05T13:31:13.458Z [INFO] 127.0.0.1:37833 :: entering follower state: follower="Node at 127.0.0.1:37833 [Follower]" leader-address= leader-id= -2024-07-05T13:31:13.458Z [WARN] 127.0.0.1:37833 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= ---- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.16s) +2024-07-05T13:37:30.830Z [INFO] 127.0.0.1:42649 :: starting node: addr=127.0.0.1:42649 +2024-07-05T13:37:30.831Z [INFO] 127.0.0.1:42649 :: initial configuration: index=0 servers=[] +2024-07-05T13:37:30.831Z [INFO] 127.0.0.1:42649 :: entering follower state: follower="Node at 127.0.0.1:42649 [Follower]" leader-address= leader-id= +2024-07-05T13:37:30.831Z [WARN] 127.0.0.1:42649 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +--- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s) === RUN TestSaturationMetric === RUN TestSaturationMetric/without_smoothing --- PASS: TestSaturationMetric (0.00s) @@ -4163,9 +4229,9 @@ === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout ---- PASS: TestRandomTimeout (0.01s) +--- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed ---- PASS: TestNewSeed (0.00s) +--- PASS: TestNewSeed (0.01s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin @@ -4173,13 +4239,13 @@ === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID ---- PASS: TestGenerateUUID (0.01s) +--- PASS: TestGenerateUUID (0.03s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) === RUN TestOverrideNotifyBool --- PASS: TestOverrideNotifyBool (0.00s) PASS -ok github.com/hashicorp/raft 114.423s +ok github.com/hashicorp/raft 117.040s make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' create-stamp debian/debhelper-build-stamp dh_testroot -O--buildsystem=golang @@ -4207,12 +4273,14 @@ dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration +I: user script /srv/workspace/pbuilder/22201/tmp/hooks/B01_cleanup starting +I: user script /srv/workspace/pbuilder/22201/tmp/hooks/B01_cleanup finished I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env -I: removing directory /srv/workspace/pbuilder/18672 and its subdirectories -I: Current time: Fri Jul 5 01:31:24 -12 2024 -I: pbuilder-time-stamp: 1720186284 +I: removing directory /srv/workspace/pbuilder/22201 and its subdirectories +I: Current time: Sat Jul 6 03:37:47 +14 2024 +I: pbuilder-time-stamp: 1720186667