Wed Aug 25 09:25:53 UTC 2021 I: starting to build golang-github-hashicorp-raft/bullseye/armhf on jenkins on '2021-08-25 09:25' Wed Aug 25 09:25:53 UTC 2021 I: The jenkins build log is/was available at https://jenkins.debian.net/userContent/reproducible/debian/build_service/armhf_6/5445/console.log Wed Aug 25 09:25:53 UTC 2021 I: Downloading source for bullseye/golang-github-hashicorp-raft=1.1.2-1 --2021-08-25 09:25:53-- http://cdn-fastly.deb.debian.org/debian/pool/main/g/golang-github-hashicorp-raft/golang-github-hashicorp-raft_1.1.2-1.dsc Connecting to 78.137.99.97:3128... connected. Proxy request sent, awaiting response... 200 OK Length: 2408 (2.4K) Saving to: ‘golang-github-hashicorp-raft_1.1.2-1.dsc’ 0K .. 100% 313M=0s 2021-08-25 09:25:53 (313 MB/s) - ‘golang-github-hashicorp-raft_1.1.2-1.dsc’ saved [2408/2408] Wed Aug 25 09:25:53 UTC 2021 I: golang-github-hashicorp-raft_1.1.2-1.dsc -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Format: 3.0 (quilt) Source: golang-github-hashicorp-raft Binary: golang-github-hashicorp-raft-dev Architecture: all Version: 1.1.2-1 Maintainer: Debian Go Packaging Team Uploaders: Tim Potter , Dmitry Smirnov , Homepage: https://github.com/hashicorp/raft Standards-Version: 4.4.1 Vcs-Browser: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft Vcs-Git: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft.git Testsuite: autopkgtest-pkg-go Build-Depends: debhelper (>= 12~), dh-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~) Package-List: golang-github-hashicorp-raft-dev deb devel optional arch=all Checksums-Sha1: 6eb88916d5b8568cc5e49dfd778e53f425f02612 103160 golang-github-hashicorp-raft_1.1.2.orig.tar.xz 3efa08a30605d234327a7c1293b36ecde63ffbbe 8980 golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz Checksums-Sha256: 6c3383ee430aa6b421cc4cb4cbc485b0e612bda2e94e88d416d13d65d375dc89 103160 golang-github-hashicorp-raft_1.1.2.orig.tar.xz 3877bf052d5719266221189613502262d1be1af152c736d3dc8ed49734ad6dd8 8980 golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz Files: 65844d5b08afa2d62ea4dedaf43e6047 103160 golang-github-hashicorp-raft_1.1.2.orig.tar.xz 036528358012213daa0588e86be2a3d4 8980 golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz Go-Import-Path: github.com/hashicorp/raft -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEULx8+TnSDCcqawZWUra72VOWjRsFAl4kBCwACgkQUra72VOW jRsPQBAAxzZChwaTUFMUIQz9GgdwyR5o2SjXw48JJ//lfygslStaiz8ju52lp33x a1d56obRwTy2QmY2s6+oV3Mq4sQWs7U/1PGnywKeA71Us10A4AAv5d9wMRq1eqzp PcyVdhfM0zFcmJf8Wr93Mi8wbCr9PT+oTxc59+zWDruS7xCoe3aDtaekVfb8QZyj OZPsaTtneSxKjqk+V5Jyx8X6Y6P2cwv89GDd4nxTxapw8H38e2cBk7BsYmXUNXye 6N7dOChDN+x6gqgBU0O97oex+QOpd4LNoFBl2gEi7kukw+jrjrdyKQCANPh0CKx0 K4MPq2BeI3qKJV21l2z7S30BSjduvyktC7efckw5CX4zgT9/kJ5e7ZAf/og+M2vc CoCV+aoAH1OOWbkPu0dsuJiA5z1ZkKGa8ZB5MgvUentk7LVd/HaXW9QDdRTNZT8z Jdh6/wnpSn8ZZEhAixTtqNt64s0sNWmh4krF0WgGNZ1djzYd6gVI1LqlJTaSRKFo nxGsEe+jnC6kv2u8XpLG+xevTKcu4UIFHq2ebpanFvxzqxAv1dWTBDDI13NApMZb Po5huhcdLapUW4NGW/3pkdJkpcY5u26zH6YvTxDICU/tHXDo5/BqI9E1jEj+5m5J XrNqpKLFjW2I9gRcmT5vy3O4edLCPBFs6MGQBK7u1V+nEkh8pJc= =IObb -----END PGP SIGNATURE----- Wed Aug 25 09:25:53 UTC 2021 I: Checking whether the package is not for us Wed Aug 25 09:25:53 UTC 2021 I: Starting 1st build on remote node virt64b-armhf-rb.debian.net. Wed Aug 25 09:25:53 UTC 2021 I: Preparing to do remote build '1' on virt64b-armhf-rb.debian.net. Wed Aug 25 09:28:45 UTC 2021 I: Deleting $TMPDIR on virt64b-armhf-rb.debian.net. I: pbuilder: network access will be disabled during build I: Current time: Tue Aug 24 21:26:03 -12 2021 I: pbuilder-time-stamp: 1629883563 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/bullseye-reproducible-base.tgz] I: copying local configuration I: mounting /proc filesystem I: mounting /sys filesystem I: creating /{dev,run}/shm I: mounting /dev/pts filesystem I: redirecting /dev/ptmx to /dev/pts/ptmx I: policy-rc.d already exists I: Copying source file I: copying [golang-github-hashicorp-raft_1.1.2-1.dsc] I: copying [./golang-github-hashicorp-raft_1.1.2.orig.tar.xz] I: copying [./golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz] I: Extracting source gpgv: unknown type of key resource 'trustedkeys.kbx' gpgv: keyblock resource '/tmp/dpkg-verify-sig.gIqeacWY/trustedkeys.kbx': General error gpgv: Signature made Sat Jan 18 19:24:28 2020 -12 gpgv: using RSA key 50BC7CF939D20C272A6B065652B6BBD953968D1B gpgv: Can't check signature: No public key dpkg-source: warning: failed to verify signature on ./golang-github-hashicorp-raft_1.1.2-1.dsc dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.1.2 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2.orig.tar.xz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying t-skip--TestNetworkTransport_AppendEntriesPipeline_CloseStreams.patch dpkg-source: info: applying t-skip--TestRaft_AutoSnapshot.patch dpkg-source: info: applying t-skip--TestRaft_LeadershipTransferLeaderRejectsClientRequests.patch dpkg-source: info: applying t-skip--TestRaft_RecoverCluster.patch dpkg-source: info: applying t-skip--TestRaft_SnapshotRestore_PeerChange.patch I: using fakeroot in build. I: Installing the build-deps I: user script /srv/workspace/pbuilder/23540/tmp/hooks/D02_print_environment starting I: set BUILDDIR='/build' BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' BUILDUSERNAME='pbuilder1' BUILD_ARCH='armhf' DEBIAN_FRONTEND='noninteractive' DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all,-fixfilepath parallel=3' DISTRIBUTION='' HOME='/root' HOST_ARCH='armhf' IFS=' ' INVOCATION_ID='6c7596e6fc9146c585771f224c323c99' LANG='C' LANGUAGE='en_US:en' LC_ALL='C' MAIL='/var/mail/root' OPTIND='1' PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' PBCURRENTCOMMANDLINEOPERATION='build' PBUILDER_OPERATION='build' PBUILDER_PKGDATADIR='/usr/share/pbuilder' PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' PBUILDER_SYSCONFDIR='/etc' PPID='23540' PS1='# ' PS2='> ' PS4='+ ' PWD='/' SHELL='/bin/bash' SHLVL='2' SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/tmp.wjmIrRuTSj/pbuilderrc_UzKj --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bullseye-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/tmp.wjmIrRuTSj/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.1.2-1.dsc' SUDO_GID='113' SUDO_UID='107' SUDO_USER='jenkins' TERM='unknown' TZ='/usr/share/zoneinfo/Etc/GMT+12' USER='root' _='/usr/bin/systemd-run' http_proxy='http://10.0.0.15:8000/' I: uname -a Linux virt64b 5.10.0-8-arm64 #1 SMP Debian 5.10.46-4 (2021-08-03) aarch64 GNU/Linux I: ls -l /bin total 3580 -rwxr-xr-x 1 root root 816764 Aug 4 08:25 bash -rwxr-xr-x 3 root root 26052 Jul 20 2020 bunzip2 -rwxr-xr-x 3 root root 26052 Jul 20 2020 bzcat lrwxrwxrwx 1 root root 6 Jul 20 2020 bzcmp -> bzdiff -rwxr-xr-x 1 root root 2225 Jul 20 2020 bzdiff lrwxrwxrwx 1 root root 6 Jul 20 2020 bzegrep -> bzgrep -rwxr-xr-x 1 root root 4877 Sep 4 2019 bzexe lrwxrwxrwx 1 root root 6 Jul 20 2020 bzfgrep -> bzgrep -rwxr-xr-x 1 root root 3775 Jul 20 2020 bzgrep -rwxr-xr-x 3 root root 26052 Jul 20 2020 bzip2 -rwxr-xr-x 1 root root 9636 Jul 20 2020 bzip2recover lrwxrwxrwx 1 root root 6 Jul 20 2020 bzless -> bzmore -rwxr-xr-x 1 root root 1297 Jul 20 2020 bzmore -rwxr-xr-x 1 root root 26668 Sep 22 2020 cat -rwxr-xr-x 1 root root 43104 Sep 22 2020 chgrp -rwxr-xr-x 1 root root 38984 Sep 22 2020 chmod -rwxr-xr-x 1 root root 43112 Sep 22 2020 chown -rwxr-xr-x 1 root root 92616 Sep 22 2020 cp -rwxr-xr-x 1 root root 75524 Dec 10 2020 dash -rwxr-xr-x 1 root root 75880 Sep 22 2020 date -rwxr-xr-x 1 root root 55436 Sep 22 2020 dd -rwxr-xr-x 1 root root 59912 Sep 22 2020 df -rwxr-xr-x 1 root root 96764 Sep 22 2020 dir -rwxr-xr-x 1 root root 55012 Jul 28 07:09 dmesg lrwxrwxrwx 1 root root 8 Nov 6 2019 dnsdomainname -> hostname lrwxrwxrwx 1 root root 8 Nov 6 2019 domainname -> hostname -rwxr-xr-x 1 root root 22508 Sep 22 2020 echo -rwxr-xr-x 1 root root 28 Nov 9 2020 egrep -rwxr-xr-x 1 root root 22496 Sep 22 2020 false -rwxr-xr-x 1 root root 28 Nov 9 2020 fgrep -rwxr-xr-x 1 root root 47492 Jul 28 07:09 findmnt -rwsr-xr-x 1 root root 26076 Feb 26 04:12 fusermount -rwxr-xr-x 1 root root 124508 Nov 9 2020 grep -rwxr-xr-x 2 root root 2346 Mar 2 11:30 gunzip -rwxr-xr-x 1 root root 6376 Mar 2 11:30 gzexe -rwxr-xr-x 1 root root 64212 Mar 2 11:30 gzip -rwxr-xr-x 1 root root 13784 Nov 6 2019 hostname -rwxr-xr-x 1 root root 43180 Sep 22 2020 ln -rwxr-xr-x 1 root root 35068 Feb 7 2020 login -rwxr-xr-x 1 root root 96764 Sep 22 2020 ls -rwxr-xr-x 1 root root 99940 Jul 28 07:09 lsblk -rwxr-xr-x 1 root root 51408 Sep 22 2020 mkdir -rwxr-xr-x 1 root root 43184 Sep 22 2020 mknod -rwxr-xr-x 1 root root 30780 Sep 22 2020 mktemp -rwxr-xr-x 1 root root 34408 Jul 28 07:09 more -rwsr-xr-x 1 root root 34400 Jul 28 07:09 mount -rwxr-xr-x 1 root root 9824 Jul 28 07:09 mountpoint -rwxr-xr-x 1 root root 88524 Sep 22 2020 mv lrwxrwxrwx 1 root root 8 Nov 6 2019 nisdomainname -> hostname lrwxrwxrwx 1 root root 14 Apr 18 03:38 pidof -> /sbin/killall5 -rwxr-xr-x 1 root root 26652 Sep 22 2020 pwd lrwxrwxrwx 1 root root 4 Aug 4 08:25 rbash -> bash -rwxr-xr-x 1 root root 30740 Sep 22 2020 readlink -rwxr-xr-x 1 root root 43104 Sep 22 2020 rm -rwxr-xr-x 1 root root 30732 Sep 22 2020 rmdir -rwxr-xr-x 1 root root 14144 Sep 27 2020 run-parts -rwxr-xr-x 1 root root 76012 Dec 22 2018 sed lrwxrwxrwx 1 root root 4 Aug 20 21:24 sh -> dash -rwxr-xr-x 1 root root 22532 Sep 22 2020 sleep -rwxr-xr-x 1 root root 55360 Sep 22 2020 stty -rwsr-xr-x 1 root root 46704 Jul 28 07:09 su -rwxr-xr-x 1 root root 22532 Sep 22 2020 sync -rwxr-xr-x 1 root root 340872 Feb 16 2021 tar -rwxr-xr-x 1 root root 9808 Sep 27 2020 tempfile -rwxr-xr-x 1 root root 67696 Sep 22 2020 touch -rwxr-xr-x 1 root root 22496 Sep 22 2020 true -rwxr-xr-x 1 root root 9636 Feb 26 04:12 ulockmgr_server -rwsr-xr-x 1 root root 22108 Jul 28 07:09 umount -rwxr-xr-x 1 root root 22520 Sep 22 2020 uname -rwxr-xr-x 2 root root 2346 Mar 2 11:30 uncompress -rwxr-xr-x 1 root root 96764 Sep 22 2020 vdir -rwxr-xr-x 1 root root 38512 Jul 28 07:09 wdctl lrwxrwxrwx 1 root root 8 Nov 6 2019 ypdomainname -> hostname -rwxr-xr-x 1 root root 1984 Mar 2 11:30 zcat -rwxr-xr-x 1 root root 1678 Mar 2 11:30 zcmp -rwxr-xr-x 1 root root 5880 Mar 2 11:30 zdiff -rwxr-xr-x 1 root root 29 Mar 2 11:30 zegrep -rwxr-xr-x 1 root root 29 Mar 2 11:30 zfgrep -rwxr-xr-x 1 root root 2081 Mar 2 11:30 zforce -rwxr-xr-x 1 root root 7585 Mar 2 11:30 zgrep -rwxr-xr-x 1 root root 2206 Mar 2 11:30 zless -rwxr-xr-x 1 root root 1842 Mar 2 11:30 zmore -rwxr-xr-x 1 root root 4553 Mar 2 11:30 znew I: user script /srv/workspace/pbuilder/23540/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy Version: 0.invalid.0 Architecture: armhf Maintainer: Debian Pbuilder Team Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder This package was created automatically by pbuilder to satisfy the build-dependencies of the package being currently built. Depends: debhelper (>= 12~), dh-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~) dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'. Selecting previously unselected package pbuilder-satisfydepends-dummy. (Reading database ... 19398 files and directories currently installed.) Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ... Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ... dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested: pbuilder-satisfydepends-dummy depends on debhelper (>= 12~); however: Package debhelper is not installed. pbuilder-satisfydepends-dummy depends on dh-golang; however: Package dh-golang is not installed. pbuilder-satisfydepends-dummy depends on golang-any; however: Package golang-any is not installed. pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however: Package golang-github-armon-go-metrics-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however: Package golang-github-hashicorp-go-hclog-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however: Package golang-github-hashicorp-go-msgpack-dev is not installed. Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ... Reading package lists... Building dependency tree... Reading state information... Initializing package states... Writing extended state information... Building tag database... pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) The following NEW packages will be installed: autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.15-go{a} golang-1.15-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-dgrijalva-jwt-go-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-kit-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-go-stack-stack-dev{a} golang-github-golang-mock-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-opentracing-opentracing-go-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-sirupsen-logrus-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-glog-dev{a} golang-go{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-oauth2-google-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-text-dev{a} golang-golang-x-time-dev{a} golang-golang-x-xerrors-dev{a} golang-google-cloud-compute-metadata-dev{a} golang-google-genproto-dev{a} golang-google-grpc-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-goprotobuf-dev{a} golang-protobuf-extensions-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1{a} libfile-stripnondeterminism-perl{a} libicu67{a} libmagic-mgc{a} libmagic1{a} libpipeline1{a} libprotobuf-dev{a} libprotobuf-lite23{a} libprotobuf23{a} libprotoc23{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libsigsegv2{a} libsub-override-perl{a} libtool{a} libuchardet0{a} libxml2{a} m4{a} man-db{a} po-debconf{a} protobuf-compiler{a} sensible-utils{a} zlib1g-dev{a} The following packages are RECOMMENDED but will NOT be installed: curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkg-config pkgconf wget 0 packages upgraded, 113 newly installed, 0 to remove and 0 not upgraded. Need to get 91.1 MB of archives. After unpacking 528 MB will be used. Writing extended state information... Get: 1 http://deb.debian.org/debian bullseye/main armhf bsdextrautils armhf 2.36.1-8 [138 kB] Get: 2 http://deb.debian.org/debian bullseye/main armhf libuchardet0 armhf 0.0.7-1 [65.0 kB] Get: 3 http://deb.debian.org/debian bullseye/main armhf groff-base armhf 1.22.4-6 [847 kB] Get: 4 http://deb.debian.org/debian bullseye/main armhf libpipeline1 armhf 1.5.3-1 [30.1 kB] Get: 5 http://deb.debian.org/debian bullseye/main armhf man-db armhf 2.9.4-2 [1319 kB] Get: 6 http://deb.debian.org/debian bullseye/main armhf golang-golang-x-sys-dev all 0.0~git20210124.22da62e-1 [308 kB] Get: 7 http://deb.debian.org/debian bullseye/main armhf golang-github-sirupsen-logrus-dev all 1.7.0-2 [45.3 kB] Get: 8 http://deb.debian.org/debian bullseye/main armhf sensible-utils all 0.0.14 [14.8 kB] Get: 9 http://deb.debian.org/debian bullseye/main armhf libmagic-mgc armhf 1:5.39-3 [273 kB] Get: 10 http://deb.debian.org/debian bullseye/main armhf libmagic1 armhf 1:5.39-3 [117 kB] Get: 11 http://deb.debian.org/debian bullseye/main armhf file armhf 1:5.39-3 [68.1 kB] Get: 12 http://deb.debian.org/debian bullseye/main armhf gettext-base armhf 0.21-4 [171 kB] Get: 13 http://deb.debian.org/debian bullseye/main armhf libsigsegv2 armhf 2.13-1 [34.0 kB] Get: 14 http://deb.debian.org/debian bullseye/main armhf m4 armhf 1.4.18-5 [192 kB] Get: 15 http://deb.debian.org/debian bullseye/main armhf autoconf all 2.69-14 [313 kB] Get: 16 http://deb.debian.org/debian bullseye/main armhf autotools-dev all 20180224.1+nmu1 [77.1 kB] Get: 17 http://deb.debian.org/debian bullseye/main armhf automake all 1:1.16.3-2 [814 kB] Get: 18 http://deb.debian.org/debian bullseye/main armhf autopoint all 0.21-4 [510 kB] Get: 19 http://deb.debian.org/debian bullseye/main armhf libdebhelper-perl all 13.3.4 [189 kB] Get: 20 http://deb.debian.org/debian bullseye/main armhf libtool all 2.4.6-15 [513 kB] Get: 21 http://deb.debian.org/debian bullseye/main armhf dh-autoreconf all 20 [17.1 kB] Get: 22 http://deb.debian.org/debian bullseye/main armhf libarchive-zip-perl all 1.68-1 [104 kB] Get: 23 http://deb.debian.org/debian bullseye/main armhf libsub-override-perl all 0.09-2 [10.2 kB] Get: 24 http://deb.debian.org/debian bullseye/main armhf libfile-stripnondeterminism-perl all 1.12.0-1 [26.3 kB] Get: 25 http://deb.debian.org/debian bullseye/main armhf dh-strip-nondeterminism all 1.12.0-1 [15.4 kB] Get: 26 http://deb.debian.org/debian bullseye/main armhf libelf1 armhf 0.183-1 [161 kB] Get: 27 http://deb.debian.org/debian bullseye/main armhf dwz armhf 0.13+20210201-1 [179 kB] Get: 28 http://deb.debian.org/debian bullseye/main armhf libicu67 armhf 67.1-7 [8319 kB] Get: 29 http://deb.debian.org/debian bullseye/main armhf libxml2 armhf 2.9.10+dfsg-6.7 [602 kB] Get: 30 http://deb.debian.org/debian bullseye/main armhf gettext armhf 0.21-4 [1243 kB] Get: 31 http://deb.debian.org/debian bullseye/main armhf intltool-debian all 0.35.0+20060710.5 [26.8 kB] Get: 32 http://deb.debian.org/debian bullseye/main armhf po-debconf all 1.0.21+nmu1 [248 kB] Get: 33 http://deb.debian.org/debian bullseye/main armhf debhelper all 13.3.4 [1049 kB] Get: 34 http://deb.debian.org/debian bullseye/main armhf dh-golang all 1.51 [24.6 kB] Get: 35 http://deb.debian.org/debian bullseye/main armhf golang-1.15-src armhf 1.15.9-6 [13.8 MB] Get: 36 http://deb.debian.org/debian bullseye/main armhf golang-1.15-go armhf 1.15.9-6 [43.1 MB] Get: 37 http://deb.debian.org/debian bullseye/main armhf golang-src armhf 2:1.15~1 [4856 B] Get: 38 http://deb.debian.org/debian bullseye/main armhf golang-go armhf 2:1.15~1 [23.9 kB] Get: 39 http://deb.debian.org/debian bullseye/main armhf golang-any armhf 2:1.15~1 [4976 B] Get: 40 http://deb.debian.org/debian bullseye/main armhf golang-github-alecthomas-units-dev all 0.0~git20201120.1786d5e-1 [7504 B] Get: 41 http://deb.debian.org/debian bullseye/main armhf golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB] Get: 42 http://deb.debian.org/debian bullseye/main armhf golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB] Get: 43 http://deb.debian.org/debian bullseye/main armhf golang-github-mattn-go-isatty-dev all 0.0.12-1 [6472 B] Get: 44 http://deb.debian.org/debian bullseye/main armhf golang-github-mattn-go-colorable-dev all 0.1.7-1 [9936 B] Get: 45 http://deb.debian.org/debian bullseye/main armhf golang-github-fatih-color-dev all 1.7.0-1 [11.4 kB] Get: 46 http://deb.debian.org/debian bullseye/main armhf golang-github-davecgh-go-spew-dev all 1.1.1-2 [29.7 kB] Get: 47 http://deb.debian.org/debian bullseye/main armhf golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB] Get: 48 http://deb.debian.org/debian bullseye/main armhf golang-github-stretchr-objx-dev all 0.3.0-1 [25.4 kB] Get: 49 http://deb.debian.org/debian bullseye/main armhf golang-gopkg-yaml.v3-dev all 3.0.0~git20200121.a6ecf24-3 [70.5 kB] Get: 50 http://deb.debian.org/debian bullseye/main armhf golang-github-stretchr-testify-dev all 1.6.1-2 [60.4 kB] Get: 51 http://deb.debian.org/debian bullseye/main armhf golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB] Get: 52 http://deb.debian.org/debian bullseye/main armhf golang-github-hashicorp-go-retryablehttp-dev all 0.6.4-2 [17.5 kB] Get: 53 http://deb.debian.org/debian bullseye/main armhf golang-github-pkg-errors-dev all 0.9.1-1 [13.0 kB] Get: 54 http://deb.debian.org/debian bullseye/main armhf golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-2 [3744 B] Get: 55 http://deb.debian.org/debian bullseye/main armhf golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-3 [65.9 kB] Get: 56 http://deb.debian.org/debian bullseye/main armhf golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB] Get: 57 http://deb.debian.org/debian bullseye/main armhf golang-github-hashicorp-go-uuid-dev all 1.0.2-1 [8764 B] Get: 58 http://deb.debian.org/debian bullseye/main armhf golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB] Get: 59 http://deb.debian.org/debian bullseye/main armhf golang-github-hashicorp-go-immutable-radix-dev all 1.3.0-1 [25.0 kB] Get: 60 http://deb.debian.org/debian bullseye/main armhf golang-github-pascaldekloe-goe-dev all 0.1.0-2 [21.7 kB] Get: 61 http://deb.debian.org/debian bullseye/main armhf golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB] Get: 62 http://deb.debian.org/debian bullseye/main armhf golang-github-cespare-xxhash-dev all 2.1.1-1 [8748 B] Get: 63 http://deb.debian.org/debian bullseye/main armhf golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B] Get: 64 http://deb.debian.org/debian bullseye/main armhf golang-github-modern-go-reflect2-dev all 1.0.1-1 [10.6 kB] Get: 65 http://deb.debian.org/debian bullseye/main armhf golang-github-json-iterator-go-dev all 1.1.10-2 [69.4 kB] Get: 66 http://deb.debian.org/debian bullseye/main armhf zlib1g-dev armhf 1:1.2.11.dfsg-2 [185 kB] Get: 67 http://deb.debian.org/debian bullseye/main armhf libprotobuf23 armhf 3.12.4-1 [777 kB] Get: 68 http://deb.debian.org/debian bullseye/main armhf libprotobuf-lite23 armhf 3.12.4-1 [217 kB] Get: 69 http://deb.debian.org/debian bullseye/main armhf libprotobuf-dev armhf 3.12.4-1 [1192 kB] Get: 70 http://deb.debian.org/debian bullseye/main armhf libprotoc23 armhf 3.12.4-1 [704 kB] Get: 71 http://deb.debian.org/debian bullseye/main armhf protobuf-compiler armhf 3.12.4-1 [74.7 kB] Get: 72 http://deb.debian.org/debian bullseye/main armhf golang-goprotobuf-dev armhf 1.3.4-2+b6 [1334 kB] Get: 73 http://deb.debian.org/debian bullseye/main armhf golang-github-prometheus-client-model-dev all 0.2.0-2 [8476 B] Get: 74 http://deb.debian.org/debian bullseye/main armhf golang-github-dgrijalva-jwt-go-dev all 3.2.0-3 [33.6 kB] Get: 75 http://deb.debian.org/debian bullseye/main armhf golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB] Get: 76 http://deb.debian.org/debian bullseye/main armhf golang-github-go-stack-stack-dev all 1.8.0-1 [8696 B] Get: 77 http://deb.debian.org/debian bullseye/main armhf golang-github-opentracing-opentracing-go-dev all 1.2.0-1 [29.4 kB] Get: 78 http://deb.debian.org/debian bullseye/main armhf golang-golang-x-text-dev all 0.3.6-1 [3857 kB] Get: 79 http://deb.debian.org/debian bullseye/main armhf golang-golang-x-net-dev all 1:0.0+git20210119.5f4716e+dfsg-4 [659 kB] Get: 80 http://deb.debian.org/debian bullseye/main armhf golang-golang-x-time-dev all 0.0+git20200630.3af7569-1 [10.1 kB] Get: 81 http://deb.debian.org/debian bullseye/main armhf golang-github-golang-mock-dev all 1.3.1-2 [35.1 kB] Get: 82 http://deb.debian.org/debian bullseye/main armhf golang-golang-x-xerrors-dev all 0.0~git20191204.9bdfabe-1 [14.2 kB] Get: 83 http://deb.debian.org/debian bullseye/main armhf golang-github-google-go-cmp-dev all 0.5.4-1 [83.7 kB] Get: 84 http://deb.debian.org/debian bullseye/main armhf golang-glog-dev all 0.0~git20160126.23def4e-3 [17.3 kB] Get: 85 http://deb.debian.org/debian bullseye/main armhf golang-golang-x-oauth2-dev all 0.0~git20190604.0f29369-2 [31.9 kB] Get: 86 http://deb.debian.org/debian bullseye/main armhf golang-google-cloud-compute-metadata-dev all 0.56.0-1 [34.0 kB] Get: 87 http://deb.debian.org/debian bullseye/main armhf golang-golang-x-oauth2-google-dev all 0.0~git20190604.0f29369-2 [13.2 kB] Get: 88 http://deb.debian.org/debian bullseye/main armhf golang-google-genproto-dev all 0.0~git20200413.b5235f6-1 [4037 kB] Get: 89 http://deb.debian.org/debian bullseye/main armhf golang-google-grpc-dev all 1.27.1-1 [478 kB] Get: 90 http://deb.debian.org/debian bullseye/main armhf golang-github-go-kit-kit-dev all 0.6.0-3 [103 kB] Get: 91 http://deb.debian.org/debian bullseye/main armhf golang-github-julienschmidt-httprouter-dev all 1.3.0-1 [19.3 kB] Get: 92 http://deb.debian.org/debian bullseye/main armhf golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B] Get: 93 http://deb.debian.org/debian bullseye/main armhf golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-2 [14.6 kB] Get: 94 http://deb.debian.org/debian bullseye/main armhf golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-3 [42.8 kB] Get: 95 http://deb.debian.org/debian bullseye/main armhf golang-gopkg-yaml.v2-dev all 2.4.0-1 [61.3 kB] Get: 96 http://deb.debian.org/debian bullseye/main armhf golang-protobuf-extensions-dev all 1.0.1-1 [29.6 kB] Get: 97 http://deb.debian.org/debian bullseye/main armhf golang-github-prometheus-common-dev all 0.15.0-2 [102 kB] Get: 98 http://deb.debian.org/debian bullseye/main armhf golang-golang-x-sync-dev all 0.0~git20210220.036812b-1 [19.1 kB] Get: 99 http://deb.debian.org/debian bullseye/main armhf golang-github-prometheus-procfs-dev all 0.3.0-2 [110 kB] Get: 100 http://deb.debian.org/debian bullseye/main armhf golang-github-prometheus-client-golang-dev all 1.9.0-2 [134 kB] Get: 101 http://deb.debian.org/debian bullseye/main armhf golang-github-armon-go-metrics-dev all 0.3.4-1 [30.5 kB] Get: 102 http://deb.debian.org/debian bullseye/main armhf golang-github-creack-pty-dev all 1.1.11-1 [8380 B] Get: 103 http://deb.debian.org/debian bullseye/main armhf golang-github-vmihailenco-tagparser-dev all 0.1.1-2 [4440 B] Get: 104 http://deb.debian.org/debian bullseye/main armhf golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB] Get: 105 http://deb.debian.org/debian bullseye/main armhf golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B] Get: 106 http://deb.debian.org/debian bullseye/main armhf libsasl2-modules-db armhf 2.1.27+dfsg-2.1 [67.6 kB] Get: 107 http://deb.debian.org/debian bullseye/main armhf libsasl2-2 armhf 2.1.27+dfsg-2.1 [99.1 kB] Get: 108 http://deb.debian.org/debian bullseye/main armhf libsasl2-dev armhf 2.1.27+dfsg-2.1 [267 kB] Get: 109 http://deb.debian.org/debian bullseye/main armhf golang-github-kr-text-dev all 0.2.0-1 [11.1 kB] Get: 110 http://deb.debian.org/debian bullseye/main armhf golang-github-kr-pretty-dev all 0.2.1+git20200831.59b4212-1 [14.6 kB] Get: 111 http://deb.debian.org/debian bullseye/main armhf golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 112 http://deb.debian.org/debian bullseye/main armhf golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 113 http://deb.debian.org/debian bullseye/main armhf golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] Fetched 91.1 MB in 11s (7992 kB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package bsdextrautils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19398 files and directories currently installed.) Preparing to unpack .../000-bsdextrautils_2.36.1-8_armhf.deb ... Unpacking bsdextrautils (2.36.1-8) ... Selecting previously unselected package libuchardet0:armhf. Preparing to unpack .../001-libuchardet0_0.0.7-1_armhf.deb ... Unpacking libuchardet0:armhf (0.0.7-1) ... Selecting previously unselected package groff-base. Preparing to unpack .../002-groff-base_1.22.4-6_armhf.deb ... Unpacking groff-base (1.22.4-6) ... Selecting previously unselected package libpipeline1:armhf. Preparing to unpack .../003-libpipeline1_1.5.3-1_armhf.deb ... Unpacking libpipeline1:armhf (1.5.3-1) ... Selecting previously unselected package man-db. Preparing to unpack .../004-man-db_2.9.4-2_armhf.deb ... Unpacking man-db (2.9.4-2) ... Selecting previously unselected package golang-golang-x-sys-dev. Preparing to unpack .../005-golang-golang-x-sys-dev_0.0~git20210124.22da62e-1_all.deb ... Unpacking golang-golang-x-sys-dev (0.0~git20210124.22da62e-1) ... Selecting previously unselected package golang-github-sirupsen-logrus-dev. Preparing to unpack .../006-golang-github-sirupsen-logrus-dev_1.7.0-2_all.deb ... Unpacking golang-github-sirupsen-logrus-dev (1.7.0-2) ... Selecting previously unselected package sensible-utils. Preparing to unpack .../007-sensible-utils_0.0.14_all.deb ... Unpacking sensible-utils (0.0.14) ... Selecting previously unselected package libmagic-mgc. Preparing to unpack .../008-libmagic-mgc_1%3a5.39-3_armhf.deb ... Unpacking libmagic-mgc (1:5.39-3) ... Selecting previously unselected package libmagic1:armhf. Preparing to unpack .../009-libmagic1_1%3a5.39-3_armhf.deb ... Unpacking libmagic1:armhf (1:5.39-3) ... Selecting previously unselected package file. Preparing to unpack .../010-file_1%3a5.39-3_armhf.deb ... Unpacking file (1:5.39-3) ... Selecting previously unselected package gettext-base. Preparing to unpack .../011-gettext-base_0.21-4_armhf.deb ... Unpacking gettext-base (0.21-4) ... Selecting previously unselected package libsigsegv2:armhf. Preparing to unpack .../012-libsigsegv2_2.13-1_armhf.deb ... Unpacking libsigsegv2:armhf (2.13-1) ... Selecting previously unselected package m4. Preparing to unpack .../013-m4_1.4.18-5_armhf.deb ... Unpacking m4 (1.4.18-5) ... Selecting previously unselected package autoconf. Preparing to unpack .../014-autoconf_2.69-14_all.deb ... Unpacking autoconf (2.69-14) ... Selecting previously unselected package autotools-dev. Preparing to unpack .../015-autotools-dev_20180224.1+nmu1_all.deb ... Unpacking autotools-dev (20180224.1+nmu1) ... Selecting previously unselected package automake. Preparing to unpack .../016-automake_1%3a1.16.3-2_all.deb ... Unpacking automake (1:1.16.3-2) ... Selecting previously unselected package autopoint. Preparing to unpack .../017-autopoint_0.21-4_all.deb ... Unpacking autopoint (0.21-4) ... Selecting previously unselected package libdebhelper-perl. Preparing to unpack .../018-libdebhelper-perl_13.3.4_all.deb ... Unpacking libdebhelper-perl (13.3.4) ... Selecting previously unselected package libtool. Preparing to unpack .../019-libtool_2.4.6-15_all.deb ... Unpacking libtool (2.4.6-15) ... Selecting previously unselected package dh-autoreconf. Preparing to unpack .../020-dh-autoreconf_20_all.deb ... Unpacking dh-autoreconf (20) ... Selecting previously unselected package libarchive-zip-perl. Preparing to unpack .../021-libarchive-zip-perl_1.68-1_all.deb ... Unpacking libarchive-zip-perl (1.68-1) ... Selecting previously unselected package libsub-override-perl. Preparing to unpack .../022-libsub-override-perl_0.09-2_all.deb ... Unpacking libsub-override-perl (0.09-2) ... Selecting previously unselected package libfile-stripnondeterminism-perl. Preparing to unpack .../023-libfile-stripnondeterminism-perl_1.12.0-1_all.deb ... Unpacking libfile-stripnondeterminism-perl (1.12.0-1) ... Selecting previously unselected package dh-strip-nondeterminism. Preparing to unpack .../024-dh-strip-nondeterminism_1.12.0-1_all.deb ... Unpacking dh-strip-nondeterminism (1.12.0-1) ... Selecting previously unselected package libelf1:armhf. Preparing to unpack .../025-libelf1_0.183-1_armhf.deb ... Unpacking libelf1:armhf (0.183-1) ... Selecting previously unselected package dwz. Preparing to unpack .../026-dwz_0.13+20210201-1_armhf.deb ... Unpacking dwz (0.13+20210201-1) ... Selecting previously unselected package libicu67:armhf. Preparing to unpack .../027-libicu67_67.1-7_armhf.deb ... Unpacking libicu67:armhf (67.1-7) ... Selecting previously unselected package libxml2:armhf. Preparing to unpack .../028-libxml2_2.9.10+dfsg-6.7_armhf.deb ... Unpacking libxml2:armhf (2.9.10+dfsg-6.7) ... Selecting previously unselected package gettext. Preparing to unpack .../029-gettext_0.21-4_armhf.deb ... Unpacking gettext (0.21-4) ... Selecting previously unselected package intltool-debian. Preparing to unpack .../030-intltool-debian_0.35.0+20060710.5_all.deb ... Unpacking intltool-debian (0.35.0+20060710.5) ... Selecting previously unselected package po-debconf. Preparing to unpack .../031-po-debconf_1.0.21+nmu1_all.deb ... Unpacking po-debconf (1.0.21+nmu1) ... Selecting previously unselected package debhelper. Preparing to unpack .../032-debhelper_13.3.4_all.deb ... Unpacking debhelper (13.3.4) ... Selecting previously unselected package dh-golang. Preparing to unpack .../033-dh-golang_1.51_all.deb ... Unpacking dh-golang (1.51) ... Selecting previously unselected package golang-1.15-src. Preparing to unpack .../034-golang-1.15-src_1.15.9-6_armhf.deb ... Unpacking golang-1.15-src (1.15.9-6) ... Selecting previously unselected package golang-1.15-go. Preparing to unpack .../035-golang-1.15-go_1.15.9-6_armhf.deb ... Unpacking golang-1.15-go (1.15.9-6) ... Selecting previously unselected package golang-src:armhf. Preparing to unpack .../036-golang-src_2%3a1.15~1_armhf.deb ... Unpacking golang-src:armhf (2:1.15~1) ... Selecting previously unselected package golang-go. Preparing to unpack .../037-golang-go_2%3a1.15~1_armhf.deb ... Unpacking golang-go (2:1.15~1) ... Selecting previously unselected package golang-any:armhf. Preparing to unpack .../038-golang-any_2%3a1.15~1_armhf.deb ... Unpacking golang-any:armhf (2:1.15~1) ... Selecting previously unselected package golang-github-alecthomas-units-dev. Preparing to unpack .../039-golang-github-alecthomas-units-dev_0.0~git20201120.1786d5e-1_all.deb ... Unpacking golang-github-alecthomas-units-dev (0.0~git20201120.1786d5e-1) ... Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev. Preparing to unpack .../040-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ... Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev. Preparing to unpack .../041-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ... Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Selecting previously unselected package golang-github-mattn-go-isatty-dev. Preparing to unpack .../042-golang-github-mattn-go-isatty-dev_0.0.12-1_all.deb ... Unpacking golang-github-mattn-go-isatty-dev (0.0.12-1) ... Selecting previously unselected package golang-github-mattn-go-colorable-dev. Preparing to unpack .../043-golang-github-mattn-go-colorable-dev_0.1.7-1_all.deb ... Unpacking golang-github-mattn-go-colorable-dev (0.1.7-1) ... Selecting previously unselected package golang-github-fatih-color-dev. Preparing to unpack .../044-golang-github-fatih-color-dev_1.7.0-1_all.deb ... Unpacking golang-github-fatih-color-dev (1.7.0-1) ... Selecting previously unselected package golang-github-davecgh-go-spew-dev. Preparing to unpack .../045-golang-github-davecgh-go-spew-dev_1.1.1-2_all.deb ... Unpacking golang-github-davecgh-go-spew-dev (1.1.1-2) ... Selecting previously unselected package golang-github-pmezard-go-difflib-dev. Preparing to unpack .../046-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ... Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Selecting previously unselected package golang-github-stretchr-objx-dev. Preparing to unpack .../047-golang-github-stretchr-objx-dev_0.3.0-1_all.deb ... Unpacking golang-github-stretchr-objx-dev (0.3.0-1) ... Selecting previously unselected package golang-gopkg-yaml.v3-dev. Preparing to unpack .../048-golang-gopkg-yaml.v3-dev_3.0.0~git20200121.a6ecf24-3_all.deb ... Unpacking golang-gopkg-yaml.v3-dev (3.0.0~git20200121.a6ecf24-3) ... Selecting previously unselected package golang-github-stretchr-testify-dev. Preparing to unpack .../049-golang-github-stretchr-testify-dev_1.6.1-2_all.deb ... Unpacking golang-github-stretchr-testify-dev (1.6.1-2) ... Selecting previously unselected package golang-github-hashicorp-go-hclog-dev. Preparing to unpack .../050-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ... Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev. Preparing to unpack .../051-golang-github-hashicorp-go-retryablehttp-dev_0.6.4-2_all.deb ... Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.6.4-2) ... Selecting previously unselected package golang-github-pkg-errors-dev. Preparing to unpack .../052-golang-github-pkg-errors-dev_0.9.1-1_all.deb ... Unpacking golang-github-pkg-errors-dev (0.9.1-1) ... Selecting previously unselected package golang-github-tv42-httpunix-dev. Preparing to unpack .../053-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-2_all.deb ... Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-2) ... Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev. Preparing to unpack .../054-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-3_all.deb ... Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-3) ... Selecting previously unselected package golang-github-datadog-datadog-go-dev. Preparing to unpack .../055-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ... Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ... Selecting previously unselected package golang-github-hashicorp-go-uuid-dev. Preparing to unpack .../056-golang-github-hashicorp-go-uuid-dev_1.0.2-1_all.deb ... Unpacking golang-github-hashicorp-go-uuid-dev (1.0.2-1) ... Selecting previously unselected package golang-github-hashicorp-golang-lru-dev. Preparing to unpack .../057-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ... Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev. Preparing to unpack .../058-golang-github-hashicorp-go-immutable-radix-dev_1.3.0-1_all.deb ... Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.0-1) ... Selecting previously unselected package golang-github-pascaldekloe-goe-dev. Preparing to unpack .../059-golang-github-pascaldekloe-goe-dev_0.1.0-2_all.deb ... Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-2) ... Selecting previously unselected package golang-github-beorn7-perks-dev. Preparing to unpack .../060-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ... Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ... Selecting previously unselected package golang-github-cespare-xxhash-dev. Preparing to unpack .../061-golang-github-cespare-xxhash-dev_2.1.1-1_all.deb ... Unpacking golang-github-cespare-xxhash-dev (2.1.1-1) ... Selecting previously unselected package golang-github-modern-go-concurrent-dev. Preparing to unpack .../062-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ... Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Selecting previously unselected package golang-github-modern-go-reflect2-dev. Preparing to unpack .../063-golang-github-modern-go-reflect2-dev_1.0.1-1_all.deb ... Unpacking golang-github-modern-go-reflect2-dev (1.0.1-1) ... Selecting previously unselected package golang-github-json-iterator-go-dev. Preparing to unpack .../064-golang-github-json-iterator-go-dev_1.1.10-2_all.deb ... Unpacking golang-github-json-iterator-go-dev (1.1.10-2) ... Selecting previously unselected package zlib1g-dev:armhf. Preparing to unpack .../065-zlib1g-dev_1%3a1.2.11.dfsg-2_armhf.deb ... Unpacking zlib1g-dev:armhf (1:1.2.11.dfsg-2) ... Selecting previously unselected package libprotobuf23:armhf. Preparing to unpack .../066-libprotobuf23_3.12.4-1_armhf.deb ... Unpacking libprotobuf23:armhf (3.12.4-1) ... Selecting previously unselected package libprotobuf-lite23:armhf. Preparing to unpack .../067-libprotobuf-lite23_3.12.4-1_armhf.deb ... Unpacking libprotobuf-lite23:armhf (3.12.4-1) ... Selecting previously unselected package libprotobuf-dev:armhf. Preparing to unpack .../068-libprotobuf-dev_3.12.4-1_armhf.deb ... Unpacking libprotobuf-dev:armhf (3.12.4-1) ... Selecting previously unselected package libprotoc23:armhf. Preparing to unpack .../069-libprotoc23_3.12.4-1_armhf.deb ... Unpacking libprotoc23:armhf (3.12.4-1) ... Selecting previously unselected package protobuf-compiler. Preparing to unpack .../070-protobuf-compiler_3.12.4-1_armhf.deb ... Unpacking protobuf-compiler (3.12.4-1) ... Selecting previously unselected package golang-goprotobuf-dev. Preparing to unpack .../071-golang-goprotobuf-dev_1.3.4-2+b6_armhf.deb ... Unpacking golang-goprotobuf-dev (1.3.4-2+b6) ... Selecting previously unselected package golang-github-prometheus-client-model-dev. Preparing to unpack .../072-golang-github-prometheus-client-model-dev_0.2.0-2_all.deb ... Unpacking golang-github-prometheus-client-model-dev (0.2.0-2) ... Selecting previously unselected package golang-github-dgrijalva-jwt-go-dev. Preparing to unpack .../073-golang-github-dgrijalva-jwt-go-dev_3.2.0-3_all.deb ... Unpacking golang-github-dgrijalva-jwt-go-dev (3.2.0-3) ... Selecting previously unselected package golang-github-go-logfmt-logfmt-dev. Preparing to unpack .../074-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ... Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Selecting previously unselected package golang-github-go-stack-stack-dev. Preparing to unpack .../075-golang-github-go-stack-stack-dev_1.8.0-1_all.deb ... Unpacking golang-github-go-stack-stack-dev (1.8.0-1) ... Selecting previously unselected package golang-github-opentracing-opentracing-go-dev. Preparing to unpack .../076-golang-github-opentracing-opentracing-go-dev_1.2.0-1_all.deb ... Unpacking golang-github-opentracing-opentracing-go-dev (1.2.0-1) ... Selecting previously unselected package golang-golang-x-text-dev. Preparing to unpack .../077-golang-golang-x-text-dev_0.3.6-1_all.deb ... Unpacking golang-golang-x-text-dev (0.3.6-1) ... Selecting previously unselected package golang-golang-x-net-dev. Preparing to unpack .../078-golang-golang-x-net-dev_1%3a0.0+git20210119.5f4716e+dfsg-4_all.deb ... Unpacking golang-golang-x-net-dev (1:0.0+git20210119.5f4716e+dfsg-4) ... Selecting previously unselected package golang-golang-x-time-dev. Preparing to unpack .../079-golang-golang-x-time-dev_0.0+git20200630.3af7569-1_all.deb ... Unpacking golang-golang-x-time-dev (0.0+git20200630.3af7569-1) ... Selecting previously unselected package golang-github-golang-mock-dev. Preparing to unpack .../080-golang-github-golang-mock-dev_1.3.1-2_all.deb ... Unpacking golang-github-golang-mock-dev (1.3.1-2) ... Selecting previously unselected package golang-golang-x-xerrors-dev. Preparing to unpack .../081-golang-golang-x-xerrors-dev_0.0~git20191204.9bdfabe-1_all.deb ... Unpacking golang-golang-x-xerrors-dev (0.0~git20191204.9bdfabe-1) ... Selecting previously unselected package golang-github-google-go-cmp-dev. Preparing to unpack .../082-golang-github-google-go-cmp-dev_0.5.4-1_all.deb ... Unpacking golang-github-google-go-cmp-dev (0.5.4-1) ... Selecting previously unselected package golang-glog-dev. Preparing to unpack .../083-golang-glog-dev_0.0~git20160126.23def4e-3_all.deb ... Unpacking golang-glog-dev (0.0~git20160126.23def4e-3) ... Selecting previously unselected package golang-golang-x-oauth2-dev. Preparing to unpack .../084-golang-golang-x-oauth2-dev_0.0~git20190604.0f29369-2_all.deb ... Unpacking golang-golang-x-oauth2-dev (0.0~git20190604.0f29369-2) ... Selecting previously unselected package golang-google-cloud-compute-metadata-dev. Preparing to unpack .../085-golang-google-cloud-compute-metadata-dev_0.56.0-1_all.deb ... Unpacking golang-google-cloud-compute-metadata-dev (0.56.0-1) ... Selecting previously unselected package golang-golang-x-oauth2-google-dev. Preparing to unpack .../086-golang-golang-x-oauth2-google-dev_0.0~git20190604.0f29369-2_all.deb ... Unpacking golang-golang-x-oauth2-google-dev (0.0~git20190604.0f29369-2) ... Selecting previously unselected package golang-google-genproto-dev. Preparing to unpack .../087-golang-google-genproto-dev_0.0~git20200413.b5235f6-1_all.deb ... Unpacking golang-google-genproto-dev (0.0~git20200413.b5235f6-1) ... Selecting previously unselected package golang-google-grpc-dev. Preparing to unpack .../088-golang-google-grpc-dev_1.27.1-1_all.deb ... Unpacking golang-google-grpc-dev (1.27.1-1) ... Selecting previously unselected package golang-github-go-kit-kit-dev. Preparing to unpack .../089-golang-github-go-kit-kit-dev_0.6.0-3_all.deb ... Unpacking golang-github-go-kit-kit-dev (0.6.0-3) ... Selecting previously unselected package golang-github-julienschmidt-httprouter-dev. Preparing to unpack .../090-golang-github-julienschmidt-httprouter-dev_1.3.0-1_all.deb ... Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-1) ... Selecting previously unselected package golang-github-jpillora-backoff-dev. Preparing to unpack .../091-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ... Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev. Preparing to unpack .../092-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-2_all.deb ... Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-2) ... Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev. Preparing to unpack .../093-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-3_all.deb ... Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-3) ... Selecting previously unselected package golang-gopkg-yaml.v2-dev. Preparing to unpack .../094-golang-gopkg-yaml.v2-dev_2.4.0-1_all.deb ... Unpacking golang-gopkg-yaml.v2-dev (2.4.0-1) ... Selecting previously unselected package golang-protobuf-extensions-dev. Preparing to unpack .../095-golang-protobuf-extensions-dev_1.0.1-1_all.deb ... Unpacking golang-protobuf-extensions-dev (1.0.1-1) ... Selecting previously unselected package golang-github-prometheus-common-dev. Preparing to unpack .../096-golang-github-prometheus-common-dev_0.15.0-2_all.deb ... Unpacking golang-github-prometheus-common-dev (0.15.0-2) ... Selecting previously unselected package golang-golang-x-sync-dev. Preparing to unpack .../097-golang-golang-x-sync-dev_0.0~git20210220.036812b-1_all.deb ... Unpacking golang-golang-x-sync-dev (0.0~git20210220.036812b-1) ... Selecting previously unselected package golang-github-prometheus-procfs-dev. Preparing to unpack .../098-golang-github-prometheus-procfs-dev_0.3.0-2_all.deb ... Unpacking golang-github-prometheus-procfs-dev (0.3.0-2) ... Selecting previously unselected package golang-github-prometheus-client-golang-dev. Preparing to unpack .../099-golang-github-prometheus-client-golang-dev_1.9.0-2_all.deb ... Unpacking golang-github-prometheus-client-golang-dev (1.9.0-2) ... Selecting previously unselected package golang-github-armon-go-metrics-dev. Preparing to unpack .../100-golang-github-armon-go-metrics-dev_0.3.4-1_all.deb ... Unpacking golang-github-armon-go-metrics-dev (0.3.4-1) ... Selecting previously unselected package golang-github-creack-pty-dev. Preparing to unpack .../101-golang-github-creack-pty-dev_1.1.11-1_all.deb ... Unpacking golang-github-creack-pty-dev (1.1.11-1) ... Selecting previously unselected package golang-github-vmihailenco-tagparser-dev. Preparing to unpack .../102-golang-github-vmihailenco-tagparser-dev_0.1.1-2_all.deb ... Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-2) ... Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev. Preparing to unpack .../103-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ... Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Selecting previously unselected package golang-gopkg-tomb.v2-dev. Preparing to unpack .../104-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ... Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Selecting previously unselected package libsasl2-modules-db:armhf. Preparing to unpack .../105-libsasl2-modules-db_2.1.27+dfsg-2.1_armhf.deb ... Unpacking libsasl2-modules-db:armhf (2.1.27+dfsg-2.1) ... Selecting previously unselected package libsasl2-2:armhf. Preparing to unpack .../106-libsasl2-2_2.1.27+dfsg-2.1_armhf.deb ... Unpacking libsasl2-2:armhf (2.1.27+dfsg-2.1) ... Selecting previously unselected package libsasl2-dev. Preparing to unpack .../107-libsasl2-dev_2.1.27+dfsg-2.1_armhf.deb ... Unpacking libsasl2-dev (2.1.27+dfsg-2.1) ... Selecting previously unselected package golang-github-kr-text-dev. Preparing to unpack .../108-golang-github-kr-text-dev_0.2.0-1_all.deb ... Unpacking golang-github-kr-text-dev (0.2.0-1) ... Selecting previously unselected package golang-github-kr-pretty-dev. Preparing to unpack .../109-golang-github-kr-pretty-dev_0.2.1+git20200831.59b4212-1_all.deb ... Unpacking golang-github-kr-pretty-dev (0.2.1+git20200831.59b4212-1) ... Selecting previously unselected package golang-gopkg-check.v1-dev. Preparing to unpack .../110-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ... Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Selecting previously unselected package golang-gopkg-mgo.v2-dev. Preparing to unpack .../111-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ... Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev. Preparing to unpack .../112-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ... Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up libpipeline1:armhf (1.5.3-1) ... Setting up golang-gopkg-yaml.v2-dev (2.4.0-1) ... Setting up golang-github-hashicorp-go-uuid-dev (1.0.2-1) ... Setting up bsdextrautils (2.36.1-8) ... update-alternatives: using /usr/bin/write.ul to provide /usr/bin/write (write) in auto mode Setting up golang-github-creack-pty-dev (1.1.11-1) ... Setting up libicu67:armhf (67.1-7) ... Setting up golang-github-golang-mock-dev (1.3.1-2) ... Setting up golang-github-stretchr-objx-dev (0.3.0-1) ... Setting up libmagic-mgc (1:5.39-3) ... Setting up golang-github-pkg-errors-dev (0.9.1-1) ... Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Setting up golang-1.15-src (1.15.9-6) ... Setting up libarchive-zip-perl (1.68-1) ... Setting up golang-gopkg-yaml.v3-dev (3.0.0~git20200121.a6ecf24-3) ... Setting up libdebhelper-perl (13.3.4) ... Setting up golang-golang-x-sys-dev (0.0~git20210124.22da62e-1) ... Setting up libmagic1:armhf (1:5.39-3) ... Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Setting up gettext-base (0.21-4) ... Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Setting up libprotobuf23:armhf (3.12.4-1) ... Setting up file (1:5.39-3) ... Setting up golang-github-go-stack-stack-dev (1.8.0-1) ... Setting up golang-github-beorn7-perks-dev (1.0.1-1) ... Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Setting up libsasl2-modules-db:armhf (2.1.27+dfsg-2.1) ... Setting up golang-github-cespare-xxhash-dev (2.1.1-1) ... Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Setting up golang-github-davecgh-go-spew-dev (1.1.1-2) ... Setting up libprotobuf-lite23:armhf (3.12.4-1) ... Setting up autotools-dev (20180224.1+nmu1) ... Setting up golang-github-pascaldekloe-goe-dev (0.1.0-2) ... Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Setting up golang-github-kr-text-dev (0.2.0-1) ... Setting up golang-github-stretchr-testify-dev (1.6.1-2) ... Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.0-1) ... Setting up libprotoc23:armhf (3.12.4-1) ... Setting up libsigsegv2:armhf (2.13-1) ... Setting up golang-golang-x-sync-dev (0.0~git20210220.036812b-1) ... Setting up autopoint (0.21-4) ... Setting up golang-glog-dev (0.0~git20160126.23def4e-3) ... Setting up libsasl2-2:armhf (2.1.27+dfsg-2.1) ... Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-1) ... Setting up zlib1g-dev:armhf (1:1.2.11.dfsg-2) ... Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-2) ... Setting up sensible-utils (0.0.14) ... Setting up golang-golang-x-text-dev (0.3.6-1) ... Setting up libuchardet0:armhf (0.0.7-1) ... Setting up golang-github-prometheus-procfs-dev (0.3.0-2) ... Setting up golang-github-alecthomas-units-dev (0.0~git20201120.1786d5e-1) ... Setting up golang-github-modern-go-reflect2-dev (1.0.1-1) ... Setting up libsub-override-perl (0.09-2) ... Setting up golang-github-dgrijalva-jwt-go-dev (3.2.0-3) ... Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ... Setting up golang-golang-x-xerrors-dev (0.0~git20191204.9bdfabe-1) ... Setting up golang-src:armhf (2:1.15~1) ... Setting up golang-1.15-go (1.15.9-6) ... Setting up libelf1:armhf (0.183-1) ... Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-2) ... Setting up libxml2:armhf (2.9.10+dfsg-6.7) ... Setting up golang-github-kr-pretty-dev (0.2.1+git20200831.59b4212-1) ... Setting up libfile-stripnondeterminism-perl (1.12.0-1) ... Setting up golang-github-google-go-cmp-dev (0.5.4-1) ... Setting up golang-github-mattn-go-isatty-dev (0.0.12-1) ... Setting up golang-github-sirupsen-logrus-dev (1.7.0-2) ... Setting up golang-github-opentracing-opentracing-go-dev (1.2.0-1) ... Setting up gettext (0.21-4) ... Setting up protobuf-compiler (3.12.4-1) ... Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-3) ... Setting up libtool (2.4.6-15) ... Setting up golang-go (2:1.15~1) ... Setting up golang-github-mattn-go-colorable-dev (0.1.7-1) ... Setting up m4 (1.4.18-5) ... Setting up golang-github-json-iterator-go-dev (1.1.10-2) ... Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Setting up golang-any:armhf (2:1.15~1) ... Setting up intltool-debian (0.35.0+20060710.5) ... Setting up libprotobuf-dev:armhf (3.12.4-1) ... Setting up libsasl2-dev (2.1.27+dfsg-2.1) ... Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Setting up golang-goprotobuf-dev (1.3.4-2+b6) ... Setting up golang-golang-x-net-dev (1:0.0+git20210119.5f4716e+dfsg-4) ... Setting up autoconf (2.69-14) ... Setting up dh-strip-nondeterminism (1.12.0-1) ... Setting up dwz (0.13+20210201-1) ... Setting up groff-base (1.22.4-6) ... Setting up golang-github-prometheus-client-model-dev (0.2.0-2) ... Setting up golang-golang-x-oauth2-dev (0.0~git20190604.0f29369-2) ... Setting up automake (1:1.16.3-2) ... update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode Setting up golang-golang-x-time-dev (0.0+git20200630.3af7569-1) ... Setting up golang-github-fatih-color-dev (1.7.0-1) ... Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up po-debconf (1.0.21+nmu1) ... Setting up golang-protobuf-extensions-dev (1.0.1-1) ... Setting up golang-google-cloud-compute-metadata-dev (0.56.0-1) ... Setting up man-db (2.9.4-2) ... Not building database; man-db/auto-update is not 'true'. Setting up dh-autoreconf (20) ... Setting up golang-golang-x-oauth2-google-dev (0.0~git20190604.0f29369-2) ... Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Setting up debhelper (13.3.4) ... Setting up golang-github-hashicorp-go-retryablehttp-dev (0.6.4-2) ... Setting up dh-golang (1.51) ... Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-3) ... Setting up golang-google-genproto-dev (0.0~git20200413.b5235f6-1) ... Setting up golang-github-prometheus-client-golang-dev (1.9.0-2) ... Setting up golang-github-armon-go-metrics-dev (0.3.4-1) ... Setting up golang-google-grpc-dev (1.27.1-1) ... Setting up golang-github-go-kit-kit-dev (0.6.0-3) ... Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-2) ... Setting up golang-github-prometheus-common-dev (0.15.0-2) ... Processing triggers for libc-bin (2.31-13) ... Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Writing extended state information... Building tag database... -> Finished parsing the build-deps Reading package lists... Building dependency tree... Reading state information... fakeroot is already the newest version (1.25.3-1.1). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. I: Building the package I: Running cd /build/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.1.2-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.1.2-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Dmitry Smirnov dpkg-source --before-build . dpkg-buildpackage: info: host architecture armhf fakeroot debian/rules clean dh clean --buildsystem=golang --with=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules build dh build --buildsystem=golang --with=golang dh_update_autotools_config -O--buildsystem=golang dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang cd obj-arm-linux-gnueabihf && go install -trimpath -v -p 3 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/cpu internal/unsafeheader runtime/internal/sys internal/race runtime/internal/math sync/atomic internal/bytealg runtime/internal/atomic unicode unicode/utf8 container/list runtime crypto/internal/subtle crypto/subtle math/bits internal/testlog math encoding unicode/utf16 internal/nettrace runtime/cgo vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/subtle golang.org/x/sys/internal/unsafeheader internal/reflectlite sync internal/singleflight math/rand errors sort io internal/oserror syscall strconv bytes strings reflect bufio time internal/syscall/unix internal/syscall/execenv github.com/hashicorp/golang-lru/simplelru github.com/hashicorp/go-immutable-radix vendor/golang.org/x/net/dns/dnsmessage context internal/poll encoding/binary internal/fmtsort os hash hash/crc32 crypto/cipher encoding/base64 crypto crypto/aes crypto/des crypto/internal/randutil crypto/sha512 crypto/ed25519/internal/edwards25519 fmt net crypto/hmac crypto/md5 crypto/rc4 math/big encoding/json log compress/flate crypto/rand crypto/elliptic compress/gzip encoding/asn1 crypto/ed25519 crypto/rsa vendor/golang.org/x/crypto/cryptobyte crypto/sha1 crypto/sha256 crypto/ecdsa crypto/dsa encoding/hex encoding/pem crypto/x509/pkix path/filepath net/url io/ioutil vendor/golang.org/x/crypto/chacha20 vendor/golang.org/x/crypto/poly1305 vendor/golang.org/x/crypto/curve25519 vendor/golang.org/x/crypto/chacha20poly1305 vendor/golang.org/x/crypto/hkdf vendor/golang.org/x/text/transform vendor/golang.org/x/text/unicode/bidi vendor/golang.org/x/text/unicode/norm crypto/x509 vendor/golang.org/x/text/secure/bidirule net/textproto vendor/golang.org/x/net/http2/hpack mime vendor/golang.org/x/net/idna crypto/tls mime/quotedprintable mime/multipart vendor/golang.org/x/net/http/httpguts vendor/golang.org/x/net/http/httpproxy net/http/internal path os/signal golang.org/x/sys/unix regexp/syntax regexp net/http/httptrace encoding/gob github.com/mattn/go-isatty github.com/mattn/go-colorable net/http github.com/fatih/color github.com/hashicorp/go-hclog go/token html text/template/parse hash/crc64 flag runtime/debug runtime/trace text/template testing html/template net/rpc github.com/armon/go-metrics github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench dh_auto_test -O--buildsystem=golang cd obj-arm-linux-gnueabihf && go test -vet=off -v -p 3 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir testing.go:153: 2021-08-25T09:28:00.244Z [INFO] creating new snapshot: path=/tmp/raft487955817/raft203318708/snapshots/3-10-1629883680244.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.01s) === RUN TestFileSS_CreateSnapshot testing.go:153: 2021-08-25T09:28:00.252Z [INFO] creating new snapshot: path=/tmp/raft591350147/snapshots/3-10-1629883680252.tmp testing.go:153: 2021-08-25T09:28:00.258Z [WARN] found temporary snapshot: name=3-10-1629883680252.tmp --- PASS: TestFileSS_CreateSnapshot (0.02s) === RUN TestFileSS_CancelSnapshot testing.go:153: 2021-08-25T09:28:00.274Z [INFO] creating new snapshot: path=/tmp/raft181443334/snapshots/3-10-1629883680274.tmp --- PASS: TestFileSS_CancelSnapshot (0.01s) === RUN TestFileSS_Retention testing.go:153: 2021-08-25T09:28:00.284Z [INFO] creating new snapshot: path=/tmp/raft008643757/snapshots/3-10-1629883680284.tmp testing.go:153: 2021-08-25T09:28:00.302Z [INFO] creating new snapshot: path=/tmp/raft008643757/snapshots/3-11-1629883680302.tmp testing.go:153: 2021-08-25T09:28:00.320Z [INFO] creating new snapshot: path=/tmp/raft008643757/snapshots/3-12-1629883680320.tmp testing.go:153: 2021-08-25T09:28:00.337Z [INFO] reaping snapshot: path=/tmp/raft008643757/snapshots/3-10-1629883680284 testing.go:153: 2021-08-25T09:28:00.338Z [INFO] creating new snapshot: path=/tmp/raft008643757/snapshots/3-13-1629883680338.tmp testing.go:153: 2021-08-25T09:28:00.354Z [INFO] reaping snapshot: path=/tmp/raft008643757/snapshots/3-11-1629883680302 testing.go:153: 2021-08-25T09:28:00.355Z [INFO] creating new snapshot: path=/tmp/raft008643757/snapshots/3-14-1629883680355.tmp testing.go:153: 2021-08-25T09:28:00.372Z [INFO] reaping snapshot: path=/tmp/raft008643757/snapshots/3-12-1629883680320 --- PASS: TestFileSS_Retention (0.09s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering testing.go:153: 2021-08-25T09:28:00.377Z [INFO] creating new snapshot: path=/tmp/raft290314716/snapshots/5-130350-1629883680377.tmp testing.go:153: 2021-08-25T09:28:00.395Z [INFO] creating new snapshot: path=/tmp/raft290314716/snapshots/36-204917-1629883680394.tmp --- PASS: TestFileSS_Ordering (0.04s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.00s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams net_transport_test.go:339: Disabled unreliable test --- SKIP: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.00s) === RUN TestNetworkTransport_RequestVote --- PASS: TestNetworkTransport_RequestVote (0.00s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.00s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff net_transport_test.go:718: countingWriter: 2021-08-25T09:28:00.440Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-08-25T09:28:00.445Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-08-25T09:28:00.455Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-08-25T09:28:00.475Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-08-25T09:28:00.516Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-08-25T09:28:00.596Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-08-25T09:28:00.757Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-08-25T09:28:01.077Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-08-25T09:28:01.717Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-08-25T09:28:02.718Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-08-25T09:28:03.718Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop testLoggerAdapter verbose: server-57d02664-b621-7445-0603-5d45444cde22: 2021-08-25T09:28:04.444Z [INFO] server-57d02664-b621-7445-0603-5d45444cde22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57d02664-b621-7445-0603-5d45444cde22 Address:57d02664-b621-7445-0603-5d45444cde22}]" testing.go:149: server-57d02664-b621-7445-0603-5d45444cde22: 2021-08-25T09:28:04.444Z [INFO] server-57d02664-b621-7445-0603-5d45444cde22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57d02664-b621-7445-0603-5d45444cde22 Address:57d02664-b621-7445-0603-5d45444cde22}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown testLoggerAdapter verbose: server-1b882f6c-13c3-ac45-0313-fccf176e94fa: 2021-08-25T09:28:04.453Z [INFO] server-1b882f6c-13c3-ac45-0313-fccf176e94fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b882f6c-13c3-ac45-0313-fccf176e94fa Address:1b882f6c-13c3-ac45-0313-fccf176e94fa}]" testing.go:149: server-1b882f6c-13c3-ac45-0313-fccf176e94fa: 2021-08-25T09:28:04.453Z [INFO] server-1b882f6c-13c3-ac45-0313-fccf176e94fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b882f6c-13c3-ac45-0313-fccf176e94fa Address:1b882f6c-13c3-ac45-0313-fccf176e94fa}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.455Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: initial configuration: index=0 servers=[] testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.455Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: 2021-08-25T09:28:04.456Z [INFO] server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: initial configuration: index=0 servers=[] testing.go:149: server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: 2021-08-25T09:28:04.456Z [INFO] server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-25ad4171-2b97-13ec-8ff6-73422e0b167e: 2021-08-25T09:28:04.456Z [INFO] server-25ad4171-2b97-13ec-8ff6-73422e0b167e: initial configuration: index=0 servers=[] testing.go:149: server-25ad4171-2b97-13ec-8ff6-73422e0b167e: 2021-08-25T09:28:04.456Z [INFO] server-25ad4171-2b97-13ec-8ff6-73422e0b167e: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.456Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: entering follower state: follower="Node at 63e6787b-e3aa-ada4-fa06-4d40330f7d20 [Follower]" leader= testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.456Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: entering follower state: follower="Node at 63e6787b-e3aa-ada4-fa06-4d40330f7d20 [Follower]" leader= testLoggerAdapter verbose: server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: 2021-08-25T09:28:04.456Z [INFO] server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: entering follower state: follower="Node at de11e8f1-6ded-c95f-940d-f2fe6b54457a [Follower]" leader= testing.go:149: server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: 2021-08-25T09:28:04.456Z [INFO] server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: entering follower state: follower="Node at de11e8f1-6ded-c95f-940d-f2fe6b54457a [Follower]" leader= testLoggerAdapter verbose: server-25ad4171-2b97-13ec-8ff6-73422e0b167e: 2021-08-25T09:28:04.456Z [INFO] server-25ad4171-2b97-13ec-8ff6-73422e0b167e: entering follower state: follower="Node at 25ad4171-2b97-13ec-8ff6-73422e0b167e [Follower]" leader= testing.go:149: server-25ad4171-2b97-13ec-8ff6-73422e0b167e: 2021-08-25T09:28:04.456Z [INFO] server-25ad4171-2b97-13ec-8ff6-73422e0b167e: entering follower state: follower="Node at 25ad4171-2b97-13ec-8ff6-73422e0b167e [Follower]" leader= testLoggerAdapter verbose: cluster: 2021-08-25T09:28:04.457Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:04.457Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: 2021-08-25T09:28:04.536Z [WARN] server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: no known peers, aborting election testing.go:149: server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: 2021-08-25T09:28:04.536Z [WARN] server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: no known peers, aborting election testLoggerAdapter verbose: server-25ad4171-2b97-13ec-8ff6-73422e0b167e: 2021-08-25T09:28:04.536Z [WARN] server-25ad4171-2b97-13ec-8ff6-73422e0b167e: no known peers, aborting election testing.go:149: server-25ad4171-2b97-13ec-8ff6-73422e0b167e: 2021-08-25T09:28:04.536Z [WARN] server-25ad4171-2b97-13ec-8ff6-73422e0b167e: no known peers, aborting election testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.539Z [WARN] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.539Z [WARN] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.539Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: entering candidate state: node="Node at 63e6787b-e3aa-ada4-fa06-4d40330f7d20 [Candidate]" term=2 testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.539Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: entering candidate state: node="Node at 63e6787b-e3aa-ada4-fa06-4d40330f7d20 [Candidate]" term=2 testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: election won: tally=2 testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: election won: tally=2 testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: entering leader state: leader="Node at 63e6787b-e3aa-ada4-fa06-4d40330f7d20 [Leader]" testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: entering leader state: leader="Node at 63e6787b-e3aa-ada4-fa06-4d40330f7d20 [Leader]" testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: added peer, starting replication: peer=server-de11e8f1-6ded-c95f-940d-f2fe6b54457a testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: added peer, starting replication: peer=server-de11e8f1-6ded-c95f-940d-f2fe6b54457a testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: added peer, starting replication: peer=server-25ad4171-2b97-13ec-8ff6-73422e0b167e testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: added peer, starting replication: peer=server-25ad4171-2b97-13ec-8ff6-73422e0b167e testLoggerAdapter verbose: server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: 2021-08-25T09:28:04.540Z [WARN] server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-25ad4171-2b97-13ec-8ff6-73422e0b167e: 2021-08-25T09:28:04.540Z [WARN] server-25ad4171-2b97-13ec-8ff6-73422e0b167e: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: 2021-08-25T09:28:04.540Z [WARN] server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [WARN] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: appendEntries rejected, sending older logs: peer="{Voter server-de11e8f1-6ded-c95f-940d-f2fe6b54457a de11e8f1-6ded-c95f-940d-f2fe6b54457a}" next=1 testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [WARN] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: appendEntries rejected, sending older logs: peer="{Voter server-de11e8f1-6ded-c95f-940d-f2fe6b54457a de11e8f1-6ded-c95f-940d-f2fe6b54457a}" next=1 testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: pipelining replication: peer="{Voter server-de11e8f1-6ded-c95f-940d-f2fe6b54457a de11e8f1-6ded-c95f-940d-f2fe6b54457a}" testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: pipelining replication: peer="{Voter server-de11e8f1-6ded-c95f-940d-f2fe6b54457a de11e8f1-6ded-c95f-940d-f2fe6b54457a}" testing.go:149: server-25ad4171-2b97-13ec-8ff6-73422e0b167e: 2021-08-25T09:28:04.540Z [WARN] server-25ad4171-2b97-13ec-8ff6-73422e0b167e: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.541Z [WARN] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: appendEntries rejected, sending older logs: peer="{Voter server-25ad4171-2b97-13ec-8ff6-73422e0b167e 25ad4171-2b97-13ec-8ff6-73422e0b167e}" next=1 testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.541Z [WARN] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: appendEntries rejected, sending older logs: peer="{Voter server-25ad4171-2b97-13ec-8ff6-73422e0b167e 25ad4171-2b97-13ec-8ff6-73422e0b167e}" next=1 testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.541Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: pipelining replication: peer="{Voter server-25ad4171-2b97-13ec-8ff6-73422e0b167e 25ad4171-2b97-13ec-8ff6-73422e0b167e}" testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.541Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: pipelining replication: peer="{Voter server-25ad4171-2b97-13ec-8ff6-73422e0b167e 25ad4171-2b97-13ec-8ff6-73422e0b167e}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:04.645Z [INFO] stable state for Follower reached at 2021-08-25 09:28:04.540230192 +0000 UTC m=+4.304446081 (2 nodes), 83.050301ms from start of poll, 84.361879ms from cluster start. Timeout at 2021-08-25 09:28:04.645406177 +0000 UTC m=+4.409622106, 105.176025ms after stability testing.go:149: cluster: 2021-08-25T09:28:04.645Z [INFO] stable state for Follower reached at 2021-08-25 09:28:04.540230192 +0000 UTC m=+4.304446081 (2 nodes), 83.050301ms from start of poll, 84.361879ms from cluster start. Timeout at 2021-08-25 09:28:04.645406177 +0000 UTC m=+4.409622106, 105.176025ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:04.645Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:04.645Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:04.751Z [INFO] stable state for Leader reached at 2021-08-25 09:28:04.645773343 +0000 UTC m=+4.409989232 (1 nodes), 2.28µs from start of poll, 189.90503ms from cluster start. Timeout at 2021-08-25 09:28:04.750904772 +0000 UTC m=+4.515120741, 105.131509ms after stability testing.go:149: cluster: 2021-08-25T09:28:04.751Z [INFO] stable state for Leader reached at 2021-08-25 09:28:04.645773343 +0000 UTC m=+4.409989232 (1 nodes), 2.28µs from start of poll, 189.90503ms from cluster start. Timeout at 2021-08-25 09:28:04.750904772 +0000 UTC m=+4.515120741, 105.131509ms after stability testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.761Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: aborting pipeline replication: peer="{Voter server-25ad4171-2b97-13ec-8ff6-73422e0b167e 25ad4171-2b97-13ec-8ff6-73422e0b167e}" testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.761Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: aborting pipeline replication: peer="{Voter server-25ad4171-2b97-13ec-8ff6-73422e0b167e 25ad4171-2b97-13ec-8ff6-73422e0b167e}" testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.761Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: aborting pipeline replication: peer="{Voter server-de11e8f1-6ded-c95f-940d-f2fe6b54457a de11e8f1-6ded-c95f-940d-f2fe6b54457a}" testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.761Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: aborting pipeline replication: peer="{Voter server-de11e8f1-6ded-c95f-940d-f2fe6b54457a de11e8f1-6ded-c95f-940d-f2fe6b54457a}" --- PASS: TestRaft_LiveBootstrap (0.31s) === RUN TestRaft_RecoverCluster_NoState testLoggerAdapter verbose: server-c7ae3f63-8e42-8412-14f1-ae6215e494ec: 2021-08-25T09:28:04.764Z [INFO] server-c7ae3f63-8e42-8412-14f1-ae6215e494ec: initial configuration: index=0 servers=[] testing.go:149: server-c7ae3f63-8e42-8412-14f1-ae6215e494ec: 2021-08-25T09:28:04.764Z [INFO] server-c7ae3f63-8e42-8412-14f1-ae6215e494ec: initial configuration: index=0 servers=[] --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster raft_test.go:119: Disabled unreliable test --- SKIP: TestRaft_RecoverCluster (0.00s) === RUN TestRaft_HasExistingState testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.766Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c134af69-1f82-2666-538f-df2b135d8202 Address:c134af69-1f82-2666-538f-df2b135d8202} {Suffrage:Voter ID:server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 Address:caf1f50e-e306-16d6-df1e-1511f7c5cfa5}]" testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.766Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c134af69-1f82-2666-538f-df2b135d8202 Address:c134af69-1f82-2666-538f-df2b135d8202} {Suffrage:Voter ID:server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 Address:caf1f50e-e306-16d6-df1e-1511f7c5cfa5}]" testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.767Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: entering follower state: follower="Node at c134af69-1f82-2666-538f-df2b135d8202 [Follower]" leader= testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.767Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: entering follower state: follower="Node at c134af69-1f82-2666-538f-df2b135d8202 [Follower]" leader= testLoggerAdapter verbose: server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5: 2021-08-25T09:28:04.767Z [INFO] server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c134af69-1f82-2666-538f-df2b135d8202 Address:c134af69-1f82-2666-538f-df2b135d8202} {Suffrage:Voter ID:server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 Address:caf1f50e-e306-16d6-df1e-1511f7c5cfa5}]" testing.go:149: server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5: 2021-08-25T09:28:04.767Z [INFO] server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c134af69-1f82-2666-538f-df2b135d8202 Address:c134af69-1f82-2666-538f-df2b135d8202} {Suffrage:Voter ID:server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 Address:caf1f50e-e306-16d6-df1e-1511f7c5cfa5}]" testLoggerAdapter verbose: server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5: 2021-08-25T09:28:04.767Z [INFO] server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5: entering follower state: follower="Node at caf1f50e-e306-16d6-df1e-1511f7c5cfa5 [Follower]" leader= testing.go:149: server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5: 2021-08-25T09:28:04.767Z [INFO] server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5: entering follower state: follower="Node at caf1f50e-e306-16d6-df1e-1511f7c5cfa5 [Follower]" leader= testLoggerAdapter verbose: server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: 2021-08-25T09:28:04.767Z [INFO] server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: initial configuration: index=0 servers=[] testing.go:149: server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: 2021-08-25T09:28:04.767Z [INFO] server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2021-08-25T09:28:04.768Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:04.768Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: 2021-08-25T09:28:04.768Z [INFO] server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: entering follower state: follower="Node at e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 [Follower]" leader= testing.go:149: server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: 2021-08-25T09:28:04.768Z [INFO] server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: entering follower state: follower="Node at e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 [Follower]" leader= testLoggerAdapter verbose: server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: 2021-08-25T09:28:04.826Z [WARN] server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: no known peers, aborting election testing.go:149: server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: 2021-08-25T09:28:04.826Z [WARN] server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: no known peers, aborting election testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.830Z [WARN] server-c134af69-1f82-2666-538f-df2b135d8202: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.830Z [WARN] server-c134af69-1f82-2666-538f-df2b135d8202: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.830Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: entering candidate state: node="Node at c134af69-1f82-2666-538f-df2b135d8202 [Candidate]" term=2 testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.830Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: entering candidate state: node="Node at c134af69-1f82-2666-538f-df2b135d8202 [Candidate]" term=2 testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.831Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: election won: tally=2 testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.831Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: election won: tally=2 testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.831Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: entering leader state: leader="Node at c134af69-1f82-2666-538f-df2b135d8202 [Leader]" testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.831Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: entering leader state: leader="Node at c134af69-1f82-2666-538f-df2b135d8202 [Leader]" testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.831Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: added peer, starting replication: peer=server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.831Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: added peer, starting replication: peer=server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.831Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: pipelining replication: peer="{Voter server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 caf1f50e-e306-16d6-df1e-1511f7c5cfa5}" testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.831Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: pipelining replication: peer="{Voter server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 caf1f50e-e306-16d6-df1e-1511f7c5cfa5}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:04.936Z [INFO] stable state for Leader reached at 2021-08-25 09:28:04.831372293 +0000 UTC m=+4.595588182 (1 nodes), 63.191907ms from start of poll, 64.649392ms from cluster start. Timeout at 2021-08-25 09:28:04.936507622 +0000 UTC m=+4.700723591, 105.135409ms after stability testing.go:149: cluster: 2021-08-25T09:28:04.936Z [INFO] stable state for Leader reached at 2021-08-25 09:28:04.831372293 +0000 UTC m=+4.595588182 (1 nodes), 63.191907ms from start of poll, 64.649392ms from cluster start. Timeout at 2021-08-25 09:28:04.936507622 +0000 UTC m=+4.700723591, 105.135409ms after stability testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.936Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: updating configuration: command=AddStaging server-id=server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 server-addr=e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 servers="[{Suffrage:Voter ID:server-c134af69-1f82-2666-538f-df2b135d8202 Address:c134af69-1f82-2666-538f-df2b135d8202} {Suffrage:Voter ID:server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 Address:caf1f50e-e306-16d6-df1e-1511f7c5cfa5} {Suffrage:Voter ID:server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 Address:e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003}]" testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.936Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: updating configuration: command=AddStaging server-id=server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 server-addr=e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 servers="[{Suffrage:Voter ID:server-c134af69-1f82-2666-538f-df2b135d8202 Address:c134af69-1f82-2666-538f-df2b135d8202} {Suffrage:Voter ID:server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 Address:caf1f50e-e306-16d6-df1e-1511f7c5cfa5} {Suffrage:Voter ID:server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 Address:e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003}]" testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.937Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: added peer, starting replication: peer=server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.937Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: added peer, starting replication: peer=server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 testLoggerAdapter verbose: server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: 2021-08-25T09:28:04.937Z [WARN] server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: 2021-08-25T09:28:04.937Z [WARN] server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.937Z [WARN] server-c134af69-1f82-2666-538f-df2b135d8202: appendEntries rejected, sending older logs: peer="{Voter server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003}" next=1 testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.937Z [WARN] server-c134af69-1f82-2666-538f-df2b135d8202: appendEntries rejected, sending older logs: peer="{Voter server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003}" next=1 testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.937Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: pipelining replication: peer="{Voter server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003}" testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.937Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: pipelining replication: peer="{Voter server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:04.937Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:04.937Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.043Z [INFO] stable state for Leader reached at 2021-08-25 09:28:04.937804001 +0000 UTC m=+4.702019890 (1 nodes), 1.5µs from start of poll, 171.0811ms from cluster start. Timeout at 2021-08-25 09:28:05.042923251 +0000 UTC m=+4.807139160, 105.11927ms after stability testing.go:149: cluster: 2021-08-25T09:28:05.043Z [INFO] stable state for Leader reached at 2021-08-25 09:28:04.937804001 +0000 UTC m=+4.702019890 (1 nodes), 1.5µs from start of poll, 171.0811ms from cluster start. Timeout at 2021-08-25 09:28:05.042923251 +0000 UTC m=+4.807139160, 105.11927ms after stability testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:05.043Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: aborting pipeline replication: peer="{Voter server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003}" testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:05.043Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: aborting pipeline replication: peer="{Voter server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003}" testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:05.043Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: aborting pipeline replication: peer="{Voter server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 caf1f50e-e306-16d6-df1e-1511f7c5cfa5}" testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:05.043Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: aborting pipeline replication: peer="{Voter server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 caf1f50e-e306-16d6-df1e-1511f7c5cfa5}" --- PASS: TestRaft_HasExistingState (0.28s) === RUN TestRaft_SingleNode testLoggerAdapter verbose: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.046Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa Address:8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa}]" testing.go:149: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.046Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa Address:8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa}]" testLoggerAdapter verbose: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.046Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: entering follower state: follower="Node at 8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa [Follower]" leader= testing.go:149: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.046Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: entering follower state: follower="Node at 8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa [Follower]" leader= testLoggerAdapter verbose: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.124Z [WARN] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.124Z [WARN] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.125Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: entering candidate state: node="Node at 8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa [Candidate]" term=2 testing.go:149: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.125Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: entering candidate state: node="Node at 8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa [Candidate]" term=2 testLoggerAdapter verbose: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.125Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: election won: tally=1 testing.go:149: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.125Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: election won: tally=1 testLoggerAdapter verbose: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.125Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: entering leader state: leader="Node at 8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa [Leader]" testing.go:149: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.125Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: entering leader state: leader="Node at 8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa [Leader]" --- PASS: TestRaft_SingleNode (0.08s) === RUN TestRaft_TripleNode testLoggerAdapter verbose: server-ea11f431-3758-3c59-4d9d-3da5a7e750c9: 2021-08-25T09:28:05.127Z [INFO] server-ea11f431-3758-3c59-4d9d-3da5a7e750c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 Address:ea11f431-3758-3c59-4d9d-3da5a7e750c9} {Suffrage:Voter ID:server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 Address:040fcb8a-9aa2-6b4a-4b67-3f480a7cd767} {Suffrage:Voter ID:server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89 Address:8cb07943-dccb-81e4-0c69-cf4ecf84bd89}]" testing.go:149: server-ea11f431-3758-3c59-4d9d-3da5a7e750c9: 2021-08-25T09:28:05.127Z [INFO] server-ea11f431-3758-3c59-4d9d-3da5a7e750c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 Address:ea11f431-3758-3c59-4d9d-3da5a7e750c9} {Suffrage:Voter ID:server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 Address:040fcb8a-9aa2-6b4a-4b67-3f480a7cd767} {Suffrage:Voter ID:server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89 Address:8cb07943-dccb-81e4-0c69-cf4ecf84bd89}]" testLoggerAdapter verbose: server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767: 2021-08-25T09:28:05.128Z [INFO] server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 Address:ea11f431-3758-3c59-4d9d-3da5a7e750c9} {Suffrage:Voter ID:server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 Address:040fcb8a-9aa2-6b4a-4b67-3f480a7cd767} {Suffrage:Voter ID:server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89 Address:8cb07943-dccb-81e4-0c69-cf4ecf84bd89}]" testing.go:149: server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767: 2021-08-25T09:28:05.128Z [INFO] server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 Address:ea11f431-3758-3c59-4d9d-3da5a7e750c9} {Suffrage:Voter ID:server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 Address:040fcb8a-9aa2-6b4a-4b67-3f480a7cd767} {Suffrage:Voter ID:server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89 Address:8cb07943-dccb-81e4-0c69-cf4ecf84bd89}]" testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.128Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 Address:ea11f431-3758-3c59-4d9d-3da5a7e750c9} {Suffrage:Voter ID:server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 Address:040fcb8a-9aa2-6b4a-4b67-3f480a7cd767} {Suffrage:Voter ID:server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89 Address:8cb07943-dccb-81e4-0c69-cf4ecf84bd89}]" testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.128Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 Address:ea11f431-3758-3c59-4d9d-3da5a7e750c9} {Suffrage:Voter ID:server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 Address:040fcb8a-9aa2-6b4a-4b67-3f480a7cd767} {Suffrage:Voter ID:server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89 Address:8cb07943-dccb-81e4-0c69-cf4ecf84bd89}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.128Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:05.128Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-ea11f431-3758-3c59-4d9d-3da5a7e750c9: 2021-08-25T09:28:05.128Z [INFO] server-ea11f431-3758-3c59-4d9d-3da5a7e750c9: entering follower state: follower="Node at ea11f431-3758-3c59-4d9d-3da5a7e750c9 [Follower]" leader= testing.go:149: server-ea11f431-3758-3c59-4d9d-3da5a7e750c9: 2021-08-25T09:28:05.128Z [INFO] server-ea11f431-3758-3c59-4d9d-3da5a7e750c9: entering follower state: follower="Node at ea11f431-3758-3c59-4d9d-3da5a7e750c9 [Follower]" leader= testLoggerAdapter verbose: server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767: 2021-08-25T09:28:05.128Z [INFO] server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767: entering follower state: follower="Node at 040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 [Follower]" leader= testing.go:149: server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767: 2021-08-25T09:28:05.128Z [INFO] server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767: entering follower state: follower="Node at 040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 [Follower]" leader= testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.128Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: entering follower state: follower="Node at 8cb07943-dccb-81e4-0c69-cf4ecf84bd89 [Follower]" leader= testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.128Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: entering follower state: follower="Node at 8cb07943-dccb-81e4-0c69-cf4ecf84bd89 [Follower]" leader= testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [WARN] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [WARN] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: entering candidate state: node="Node at 8cb07943-dccb-81e4-0c69-cf4ecf84bd89 [Candidate]" term=2 testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: entering candidate state: node="Node at 8cb07943-dccb-81e4-0c69-cf4ecf84bd89 [Candidate]" term=2 testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: election won: tally=2 testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: election won: tally=2 testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: entering leader state: leader="Node at 8cb07943-dccb-81e4-0c69-cf4ecf84bd89 [Leader]" testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: entering leader state: leader="Node at 8cb07943-dccb-81e4-0c69-cf4ecf84bd89 [Leader]" testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: added peer, starting replication: peer=server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: added peer, starting replication: peer=server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: added peer, starting replication: peer=server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: added peer, starting replication: peer=server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.182Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: pipelining replication: peer="{Voter server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 ea11f431-3758-3c59-4d9d-3da5a7e750c9}" testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.182Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: pipelining replication: peer="{Voter server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 ea11f431-3758-3c59-4d9d-3da5a7e750c9}" testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.182Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: pipelining replication: peer="{Voter server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 040fcb8a-9aa2-6b4a-4b67-3f480a7cd767}" testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.182Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: pipelining replication: peer="{Voter server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 040fcb8a-9aa2-6b4a-4b67-3f480a7cd767}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.287Z [INFO] stable state for Follower reached at 2021-08-25 09:28:05.181798484 +0000 UTC m=+4.946014393 (2 nodes), 53.097565ms from start of poll, 54.11837ms from cluster start. Timeout at 2021-08-25 09:28:05.286920694 +0000 UTC m=+5.051136603, 105.12221ms after stability testing.go:149: cluster: 2021-08-25T09:28:05.287Z [INFO] stable state for Follower reached at 2021-08-25 09:28:05.181798484 +0000 UTC m=+4.946014393 (2 nodes), 53.097565ms from start of poll, 54.11837ms from cluster start. Timeout at 2021-08-25 09:28:05.286920694 +0000 UTC m=+5.051136603, 105.12221ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.287Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:05.287Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.392Z [INFO] stable state for Leader reached at 2021-08-25 09:28:05.287208987 +0000 UTC m=+5.051424896 (1 nodes), 1.98µs from start of poll, 159.528873ms from cluster start. Timeout at 2021-08-25 09:28:05.392316878 +0000 UTC m=+5.156532867, 105.107971ms after stability testing.go:149: cluster: 2021-08-25T09:28:05.392Z [INFO] stable state for Leader reached at 2021-08-25 09:28:05.287208987 +0000 UTC m=+5.051424896 (1 nodes), 1.98µs from start of poll, 159.528873ms from cluster start. Timeout at 2021-08-25 09:28:05.392316878 +0000 UTC m=+5.156532867, 105.107971ms after stability testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.418Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: aborting pipeline replication: peer="{Voter server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 040fcb8a-9aa2-6b4a-4b67-3f480a7cd767}" testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.418Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: aborting pipeline replication: peer="{Voter server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 040fcb8a-9aa2-6b4a-4b67-3f480a7cd767}" testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.418Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: aborting pipeline replication: peer="{Voter server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 ea11f431-3758-3c59-4d9d-3da5a7e750c9}" testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.418Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: aborting pipeline replication: peer="{Voter server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 ea11f431-3758-3c59-4d9d-3da5a7e750c9}" testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.469Z [ERROR] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: failed to heartbeat to: peer=ea11f431-3758-3c59-4d9d-3da5a7e750c9 error="command timed out" testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.469Z [ERROR] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: failed to heartbeat to: peer=ea11f431-3758-3c59-4d9d-3da5a7e750c9 error="command timed out" --- PASS: TestRaft_TripleNode (0.34s) === RUN TestRaft_LeaderFail testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.473Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 Address:6a039ee3-59cc-2aa4-7ff0-507c5b9a4248} {Suffrage:Voter ID:server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 Address:2d3bdee3-711d-1243-4303-aa6a47ff05b4} {Suffrage:Voter ID:server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 Address:99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}]" testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.473Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 Address:6a039ee3-59cc-2aa4-7ff0-507c5b9a4248} {Suffrage:Voter ID:server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 Address:2d3bdee3-711d-1243-4303-aa6a47ff05b4} {Suffrage:Voter ID:server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 Address:99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}]" testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.473Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: entering follower state: follower="Node at 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 [Follower]" leader= testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.473Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: entering follower state: follower="Node at 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 [Follower]" leader= testLoggerAdapter verbose: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.474Z [INFO] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 Address:6a039ee3-59cc-2aa4-7ff0-507c5b9a4248} {Suffrage:Voter ID:server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 Address:2d3bdee3-711d-1243-4303-aa6a47ff05b4} {Suffrage:Voter ID:server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 Address:99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}]" testing.go:149: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.474Z [INFO] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 Address:6a039ee3-59cc-2aa4-7ff0-507c5b9a4248} {Suffrage:Voter ID:server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 Address:2d3bdee3-711d-1243-4303-aa6a47ff05b4} {Suffrage:Voter ID:server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 Address:99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}]" testLoggerAdapter verbose: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.474Z [INFO] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: entering follower state: follower="Node at 2d3bdee3-711d-1243-4303-aa6a47ff05b4 [Follower]" leader= testing.go:149: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.474Z [INFO] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: entering follower state: follower="Node at 2d3bdee3-711d-1243-4303-aa6a47ff05b4 [Follower]" leader= testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.474Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 Address:6a039ee3-59cc-2aa4-7ff0-507c5b9a4248} {Suffrage:Voter ID:server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 Address:2d3bdee3-711d-1243-4303-aa6a47ff05b4} {Suffrage:Voter ID:server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 Address:99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}]" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.474Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 Address:6a039ee3-59cc-2aa4-7ff0-507c5b9a4248} {Suffrage:Voter ID:server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 Address:2d3bdee3-711d-1243-4303-aa6a47ff05b4} {Suffrage:Voter ID:server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 Address:99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.474Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:05.474Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.474Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering follower state: follower="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Follower]" leader= testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.474Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering follower state: follower="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Follower]" leader= testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.528Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.528Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.528Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering candidate state: node="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Candidate]" term=2 testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.528Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering candidate state: node="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Candidate]" term=2 testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.528Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: election won: tally=2 testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.528Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: election won: tally=2 testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering leader state: leader="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Leader]" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering leader state: leader="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Leader]" testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: added peer, starting replication: peer=server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: added peer, starting replication: peer=server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: added peer, starting replication: peer=server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: added peer, starting replication: peer=server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: pipelining replication: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: pipelining replication: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: pipelining replication: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: pipelining replication: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.634Z [INFO] stable state for Follower reached at 2021-08-25 09:28:05.529745746 +0000 UTC m=+5.293961655 (2 nodes), 55.087081ms from start of poll, 56.542025ms from cluster start. Timeout at 2021-08-25 09:28:05.634871835 +0000 UTC m=+5.399087804, 105.126149ms after stability testing.go:149: cluster: 2021-08-25T09:28:05.634Z [INFO] stable state for Follower reached at 2021-08-25 09:28:05.529745746 +0000 UTC m=+5.293961655 (2 nodes), 55.087081ms from start of poll, 56.542025ms from cluster start. Timeout at 2021-08-25 09:28:05.634871835 +0000 UTC m=+5.399087804, 105.126149ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.635Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:05.635Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.740Z [INFO] stable state for Leader reached at 2021-08-25 09:28:05.635205904 +0000 UTC m=+5.399421793 (1 nodes), 2.26µs from start of poll, 162.002163ms from cluster start. Timeout at 2021-08-25 09:28:05.740207365 +0000 UTC m=+5.504423254, 105.001461ms after stability testing.go:149: cluster: 2021-08-25T09:28:05.740Z [INFO] stable state for Leader reached at 2021-08-25 09:28:05.635205904 +0000 UTC m=+5.399421793 (1 nodes), 2.26µs from start of poll, 162.002163ms from cluster start. Timeout at 2021-08-25 09:28:05.740207365 +0000 UTC m=+5.504423254, 105.001461ms after stability raft_test.go:336: [INFO] Disconnecting Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Leader] testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.752Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=2d3bdee3-711d-1243-4303-aa6a47ff05b4 error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.752Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=2d3bdee3-711d-1243-4303-aa6a47ff05b4 error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.752Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.752Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.755Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to pipeline appendEntries: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="append pipeline closed" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.755Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to pipeline appendEntries: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="append pipeline closed" testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.755Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: aborting pipeline replication: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.755Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: aborting pipeline replication: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.756Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:05.756Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.756Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to pipeline appendEntries: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="append pipeline closed" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.756Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to pipeline appendEntries: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="append pipeline closed" testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.756Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: aborting pipeline replication: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.756Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: aborting pipeline replication: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.762Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.762Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.766Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.766Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.768Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.768Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.772Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=2d3bdee3-711d-1243-4303-aa6a47ff05b4 error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.772Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=2d3bdee3-711d-1243-4303-aa6a47ff05b4 error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.781Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.781Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.786Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.786Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.788Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.788Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.790Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=2d3bdee3-711d-1243-4303-aa6a47ff05b4 error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.790Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=2d3bdee3-711d-1243-4303-aa6a47ff05b4 error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.796Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to contact: server-id=server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 time=50.107663ms testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.796Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to contact: server-id=server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 time=50.107663ms testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.801Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.801Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.801Z [WARN] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: heartbeat timeout reached, starting election: last-leader=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.801Z [WARN] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: heartbeat timeout reached, starting election: last-leader=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.801Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: entering candidate state: node="Node at 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 [Candidate]" term=3 testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.801Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: entering candidate state: node="Node at 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 [Candidate]" term=3 testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.801Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to make requestVote RPC: target="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.801Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to make requestVote RPC: target="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testLoggerAdapter verbose: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.801Z [WARN] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: rejecting vote request since we have a leader: from=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 leader=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 testing.go:149: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.801Z [WARN] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: rejecting vote request since we have a leader: from=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 leader=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.803Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.803Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.806Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to contact: server-id=server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 time=59.466153ms testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.806Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to contact: server-id=server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 time=59.466153ms testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.807Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to contact: server-id=server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 time=60.398987ms testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.807Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to contact: server-id=server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 time=60.398987ms testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.807Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to contact quorum of nodes, stepping down testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.807Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.807Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering follower state: follower="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Follower]" leader= testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.807Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering follower state: follower="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Follower]" leader= testLoggerAdapter verbose: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.808Z [WARN] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: heartbeat timeout reached, starting election: last-leader=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 testing.go:149: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.808Z [WARN] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: heartbeat timeout reached, starting election: last-leader=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 testLoggerAdapter verbose: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.808Z [INFO] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: entering candidate state: node="Node at 2d3bdee3-711d-1243-4303-aa6a47ff05b4 [Candidate]" term=3 testing.go:149: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.808Z [INFO] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: entering candidate state: node="Node at 2d3bdee3-711d-1243-4303-aa6a47ff05b4 [Candidate]" term=3 testLoggerAdapter verbose: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.808Z [ERROR] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: failed to make requestVote RPC: target="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testing.go:149: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.808Z [ERROR] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: failed to make requestVote RPC: target="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.808Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: duplicate requestVote for same term: term=3 testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.808Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.860Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.860Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.861Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering candidate state: node="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Candidate]" term=3 testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.861Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering candidate state: node="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Candidate]" term=3 testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.861Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to make requestVote RPC: target="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.861Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to make requestVote RPC: target="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.861Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to make requestVote RPC: target="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.861Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to make requestVote RPC: target="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.866Z [WARN] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: Election timeout reached, restarting election testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.866Z [WARN] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: Election timeout reached, restarting election testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.866Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: entering candidate state: node="Node at 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 [Candidate]" term=4 testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.866Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: entering candidate state: node="Node at 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 [Candidate]" term=4 testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to make requestVote RPC: target="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to make requestVote RPC: target="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testLoggerAdapter verbose: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.867Z [INFO] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: entering follower state: follower="Node at 2d3bdee3-711d-1243-4303-aa6a47ff05b4 [Follower]" leader= testing.go:149: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.867Z [INFO] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: entering follower state: follower="Node at 2d3bdee3-711d-1243-4303-aa6a47ff05b4 [Follower]" leader= testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: election won: tally=2 testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: election won: tally=2 testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: entering leader state: leader="Node at 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 [Leader]" testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: entering leader state: leader="Node at 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 [Leader]" testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: added peer, starting replication: peer=server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: added peer, starting replication: peer=server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: added peer, starting replication: peer=server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: added peer, starting replication: peer=server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: pipelining replication: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: pipelining replication: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to appendEntries to: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to appendEntries to: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.877Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to heartbeat to: peer=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.877Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to heartbeat to: peer=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.885Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to appendEntries to: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.885Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to appendEntries to: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.894Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to heartbeat to: peer=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.894Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to heartbeat to: peer=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.902Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to appendEntries to: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.902Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to appendEntries to: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.914Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to heartbeat to: peer=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.914Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to heartbeat to: peer=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.914Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: Election timeout reached, restarting election testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.914Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: Election timeout reached, restarting election testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.914Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering candidate state: node="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Candidate]" term=4 testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.914Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering candidate state: node="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Candidate]" term=4 testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.914Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to make requestVote RPC: target="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.914Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to make requestVote RPC: target="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.914Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to make requestVote RPC: target="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.914Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to make requestVote RPC: target="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.917Z [WARN] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to contact: server-id=server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 time=50.186736ms testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.917Z [WARN] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to contact: server-id=server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 time=50.186736ms testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.931Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to appendEntries to: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.931Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to appendEntries to: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.940Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to heartbeat to: peer=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.940Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to heartbeat to: peer=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.967Z [WARN] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to contact: server-id=server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 time=99.74539ms testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.967Z [WARN] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to contact: server-id=server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 time=99.74539ms testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.972Z [INFO] stable state for Leader reached at 2021-08-25 09:28:05.867415502 +0000 UTC m=+5.631631431 (1 nodes), 111.215603ms from start of poll, 394.211801ms from cluster start. Timeout at 2021-08-25 09:28:05.972487477 +0000 UTC m=+5.736703386, 105.071955ms after stability testing.go:149: cluster: 2021-08-25T09:28:05.972Z [INFO] stable state for Leader reached at 2021-08-25 09:28:05.867415502 +0000 UTC m=+5.631631431 (1 nodes), 111.215603ms from start of poll, 394.211801ms from cluster start. Timeout at 2021-08-25 09:28:05.972487477 +0000 UTC m=+5.736703386, 105.071955ms after stability raft_test.go:371: [INFO] Reconnecting Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Candidate] testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.977Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering follower state: follower="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Follower]" leader=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.977Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering follower state: follower="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Follower]" leader=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.977Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: pipelining replication: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.977Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: pipelining replication: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.982Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: aborting pipeline replication: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.982Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: aborting pipeline replication: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.982Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: aborting pipeline replication: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.982Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: aborting pipeline replication: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" --- PASS: TestRaft_LeaderFail (0.51s) === RUN TestRaft_BehindFollower testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:05.986Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7f134ac-941b-8470-0852-b7f284e3abb4 Address:e7f134ac-941b-8470-0852-b7f284e3abb4} {Suffrage:Voter ID:server-55ac66af-99a1-bfc9-072b-c3a2671898b1 Address:55ac66af-99a1-bfc9-072b-c3a2671898b1} {Suffrage:Voter ID:server-126bbe33-9f7c-2831-55a6-a34e29d66dae Address:126bbe33-9f7c-2831-55a6-a34e29d66dae}]" testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:05.986Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7f134ac-941b-8470-0852-b7f284e3abb4 Address:e7f134ac-941b-8470-0852-b7f284e3abb4} {Suffrage:Voter ID:server-55ac66af-99a1-bfc9-072b-c3a2671898b1 Address:55ac66af-99a1-bfc9-072b-c3a2671898b1} {Suffrage:Voter ID:server-126bbe33-9f7c-2831-55a6-a34e29d66dae Address:126bbe33-9f7c-2831-55a6-a34e29d66dae}]" testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:05.987Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: entering follower state: follower="Node at e7f134ac-941b-8470-0852-b7f284e3abb4 [Follower]" leader= testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:05.987Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: entering follower state: follower="Node at e7f134ac-941b-8470-0852-b7f284e3abb4 [Follower]" leader= testLoggerAdapter verbose: server-55ac66af-99a1-bfc9-072b-c3a2671898b1: 2021-08-25T09:28:05.987Z [INFO] server-55ac66af-99a1-bfc9-072b-c3a2671898b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7f134ac-941b-8470-0852-b7f284e3abb4 Address:e7f134ac-941b-8470-0852-b7f284e3abb4} {Suffrage:Voter ID:server-55ac66af-99a1-bfc9-072b-c3a2671898b1 Address:55ac66af-99a1-bfc9-072b-c3a2671898b1} {Suffrage:Voter ID:server-126bbe33-9f7c-2831-55a6-a34e29d66dae Address:126bbe33-9f7c-2831-55a6-a34e29d66dae}]" testing.go:149: server-55ac66af-99a1-bfc9-072b-c3a2671898b1: 2021-08-25T09:28:05.987Z [INFO] server-55ac66af-99a1-bfc9-072b-c3a2671898b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7f134ac-941b-8470-0852-b7f284e3abb4 Address:e7f134ac-941b-8470-0852-b7f284e3abb4} {Suffrage:Voter ID:server-55ac66af-99a1-bfc9-072b-c3a2671898b1 Address:55ac66af-99a1-bfc9-072b-c3a2671898b1} {Suffrage:Voter ID:server-126bbe33-9f7c-2831-55a6-a34e29d66dae Address:126bbe33-9f7c-2831-55a6-a34e29d66dae}]" testLoggerAdapter verbose: server-55ac66af-99a1-bfc9-072b-c3a2671898b1: 2021-08-25T09:28:05.987Z [INFO] server-55ac66af-99a1-bfc9-072b-c3a2671898b1: entering follower state: follower="Node at 55ac66af-99a1-bfc9-072b-c3a2671898b1 [Follower]" leader= testing.go:149: server-55ac66af-99a1-bfc9-072b-c3a2671898b1: 2021-08-25T09:28:05.987Z [INFO] server-55ac66af-99a1-bfc9-072b-c3a2671898b1: entering follower state: follower="Node at 55ac66af-99a1-bfc9-072b-c3a2671898b1 [Follower]" leader= testLoggerAdapter verbose: server-126bbe33-9f7c-2831-55a6-a34e29d66dae: 2021-08-25T09:28:05.987Z [INFO] server-126bbe33-9f7c-2831-55a6-a34e29d66dae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7f134ac-941b-8470-0852-b7f284e3abb4 Address:e7f134ac-941b-8470-0852-b7f284e3abb4} {Suffrage:Voter ID:server-55ac66af-99a1-bfc9-072b-c3a2671898b1 Address:55ac66af-99a1-bfc9-072b-c3a2671898b1} {Suffrage:Voter ID:server-126bbe33-9f7c-2831-55a6-a34e29d66dae Address:126bbe33-9f7c-2831-55a6-a34e29d66dae}]" testing.go:149: server-126bbe33-9f7c-2831-55a6-a34e29d66dae: 2021-08-25T09:28:05.987Z [INFO] server-126bbe33-9f7c-2831-55a6-a34e29d66dae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7f134ac-941b-8470-0852-b7f284e3abb4 Address:e7f134ac-941b-8470-0852-b7f284e3abb4} {Suffrage:Voter ID:server-55ac66af-99a1-bfc9-072b-c3a2671898b1 Address:55ac66af-99a1-bfc9-072b-c3a2671898b1} {Suffrage:Voter ID:server-126bbe33-9f7c-2831-55a6-a34e29d66dae Address:126bbe33-9f7c-2831-55a6-a34e29d66dae}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.987Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:05.987Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-126bbe33-9f7c-2831-55a6-a34e29d66dae: 2021-08-25T09:28:05.987Z [INFO] server-126bbe33-9f7c-2831-55a6-a34e29d66dae: entering follower state: follower="Node at 126bbe33-9f7c-2831-55a6-a34e29d66dae [Follower]" leader= testing.go:149: server-126bbe33-9f7c-2831-55a6-a34e29d66dae: 2021-08-25T09:28:05.987Z [INFO] server-126bbe33-9f7c-2831-55a6-a34e29d66dae: entering follower state: follower="Node at 126bbe33-9f7c-2831-55a6-a34e29d66dae [Follower]" leader= testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.041Z [WARN] server-e7f134ac-941b-8470-0852-b7f284e3abb4: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.041Z [WARN] server-e7f134ac-941b-8470-0852-b7f284e3abb4: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.041Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: entering candidate state: node="Node at e7f134ac-941b-8470-0852-b7f284e3abb4 [Candidate]" term=2 testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.041Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: entering candidate state: node="Node at e7f134ac-941b-8470-0852-b7f284e3abb4 [Candidate]" term=2 testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: election won: tally=2 testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: election won: tally=2 testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: entering leader state: leader="Node at e7f134ac-941b-8470-0852-b7f284e3abb4 [Leader]" testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: entering leader state: leader="Node at e7f134ac-941b-8470-0852-b7f284e3abb4 [Leader]" testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: added peer, starting replication: peer=server-55ac66af-99a1-bfc9-072b-c3a2671898b1 testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: added peer, starting replication: peer=server-55ac66af-99a1-bfc9-072b-c3a2671898b1 testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: added peer, starting replication: peer=server-126bbe33-9f7c-2831-55a6-a34e29d66dae testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: added peer, starting replication: peer=server-126bbe33-9f7c-2831-55a6-a34e29d66dae testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: pipelining replication: peer="{Voter server-126bbe33-9f7c-2831-55a6-a34e29d66dae 126bbe33-9f7c-2831-55a6-a34e29d66dae}" testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: pipelining replication: peer="{Voter server-126bbe33-9f7c-2831-55a6-a34e29d66dae 126bbe33-9f7c-2831-55a6-a34e29d66dae}" testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: pipelining replication: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: pipelining replication: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.147Z [INFO] stable state for Leader reached at 2021-08-25 09:28:06.042558864 +0000 UTC m=+5.806774753 (1 nodes), 54.628322ms from start of poll, 55.872667ms from cluster start. Timeout at 2021-08-25 09:28:06.147702852 +0000 UTC m=+5.911918781, 105.144028ms after stability testing.go:149: cluster: 2021-08-25T09:28:06.147Z [INFO] stable state for Leader reached at 2021-08-25 09:28:06.042558864 +0000 UTC m=+5.806774753 (1 nodes), 54.628322ms from start of poll, 55.872667ms from cluster start. Timeout at 2021-08-25 09:28:06.147702852 +0000 UTC m=+5.911918781, 105.144028ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.147Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:06.147Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.253Z [INFO] stable state for Follower reached at 2021-08-25 09:28:06.147986506 +0000 UTC m=+5.912202395 (2 nodes), 2.78µs from start of poll, 161.300309ms from cluster start. Timeout at 2021-08-25 09:28:06.253147772 +0000 UTC m=+6.017363761, 105.161366ms after stability testing.go:149: cluster: 2021-08-25T09:28:06.253Z [INFO] stable state for Follower reached at 2021-08-25 09:28:06.147986506 +0000 UTC m=+5.912202395 (2 nodes), 2.78µs from start of poll, 161.300309ms from cluster start. Timeout at 2021-08-25 09:28:06.253147772 +0000 UTC m=+6.017363761, 105.161366ms after stability testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.253Z [ERROR] server-e7f134ac-941b-8470-0852-b7f284e3abb4: failed to pipeline appendEntries: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" error="append pipeline closed" testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.253Z [ERROR] server-e7f134ac-941b-8470-0852-b7f284e3abb4: failed to pipeline appendEntries: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" error="append pipeline closed" testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.253Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: aborting pipeline replication: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.253Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: aborting pipeline replication: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.253Z [ERROR] server-e7f134ac-941b-8470-0852-b7f284e3abb4: failed to appendEntries to: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" error="failed to connect to peer: 55ac66af-99a1-bfc9-072b-c3a2671898b1" testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.253Z [ERROR] server-e7f134ac-941b-8470-0852-b7f284e3abb4: failed to appendEntries to: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" error="failed to connect to peer: 55ac66af-99a1-bfc9-072b-c3a2671898b1" raft_test.go:420: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.263Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: pipelining replication: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.263Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: pipelining replication: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.277Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:06.277Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.382Z [INFO] stable state for Leader reached at 2021-08-25 09:28:06.277147002 +0000 UTC m=+6.041362891 (1 nodes), 2.16µs from start of poll, 290.460805ms from cluster start. Timeout at 2021-08-25 09:28:06.382257253 +0000 UTC m=+6.146473142, 105.110251ms after stability testing.go:149: cluster: 2021-08-25T09:28:06.382Z [INFO] stable state for Leader reached at 2021-08-25 09:28:06.277147002 +0000 UTC m=+6.041362891 (1 nodes), 2.16µs from start of poll, 290.460805ms from cluster start. Timeout at 2021-08-25 09:28:06.382257253 +0000 UTC m=+6.146473142, 105.110251ms after stability testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.382Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: aborting pipeline replication: peer="{Voter server-126bbe33-9f7c-2831-55a6-a34e29d66dae 126bbe33-9f7c-2831-55a6-a34e29d66dae}" testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.382Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: aborting pipeline replication: peer="{Voter server-126bbe33-9f7c-2831-55a6-a34e29d66dae 126bbe33-9f7c-2831-55a6-a34e29d66dae}" testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.382Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: aborting pipeline replication: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.382Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: aborting pipeline replication: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" --- PASS: TestRaft_BehindFollower (0.40s) === RUN TestRaft_ApplyNonLeader testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.386Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27343939-c938-1f03-3869-4d341d79d44b Address:27343939-c938-1f03-3869-4d341d79d44b} {Suffrage:Voter ID:server-e962818f-1d0a-0cb1-423c-a122388f9ed5 Address:e962818f-1d0a-0cb1-423c-a122388f9ed5} {Suffrage:Voter ID:server-227f3f58-3e33-e7b6-2932-9026b2d38170 Address:227f3f58-3e33-e7b6-2932-9026b2d38170}]" testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.386Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27343939-c938-1f03-3869-4d341d79d44b Address:27343939-c938-1f03-3869-4d341d79d44b} {Suffrage:Voter ID:server-e962818f-1d0a-0cb1-423c-a122388f9ed5 Address:e962818f-1d0a-0cb1-423c-a122388f9ed5} {Suffrage:Voter ID:server-227f3f58-3e33-e7b6-2932-9026b2d38170 Address:227f3f58-3e33-e7b6-2932-9026b2d38170}]" testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.386Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: entering follower state: follower="Node at 27343939-c938-1f03-3869-4d341d79d44b [Follower]" leader= testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.386Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: entering follower state: follower="Node at 27343939-c938-1f03-3869-4d341d79d44b [Follower]" leader= testLoggerAdapter verbose: server-e962818f-1d0a-0cb1-423c-a122388f9ed5: 2021-08-25T09:28:06.386Z [INFO] server-e962818f-1d0a-0cb1-423c-a122388f9ed5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27343939-c938-1f03-3869-4d341d79d44b Address:27343939-c938-1f03-3869-4d341d79d44b} {Suffrage:Voter ID:server-e962818f-1d0a-0cb1-423c-a122388f9ed5 Address:e962818f-1d0a-0cb1-423c-a122388f9ed5} {Suffrage:Voter ID:server-227f3f58-3e33-e7b6-2932-9026b2d38170 Address:227f3f58-3e33-e7b6-2932-9026b2d38170}]" testing.go:149: server-e962818f-1d0a-0cb1-423c-a122388f9ed5: 2021-08-25T09:28:06.386Z [INFO] server-e962818f-1d0a-0cb1-423c-a122388f9ed5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27343939-c938-1f03-3869-4d341d79d44b Address:27343939-c938-1f03-3869-4d341d79d44b} {Suffrage:Voter ID:server-e962818f-1d0a-0cb1-423c-a122388f9ed5 Address:e962818f-1d0a-0cb1-423c-a122388f9ed5} {Suffrage:Voter ID:server-227f3f58-3e33-e7b6-2932-9026b2d38170 Address:227f3f58-3e33-e7b6-2932-9026b2d38170}]" testLoggerAdapter verbose: server-e962818f-1d0a-0cb1-423c-a122388f9ed5: 2021-08-25T09:28:06.386Z [INFO] server-e962818f-1d0a-0cb1-423c-a122388f9ed5: entering follower state: follower="Node at e962818f-1d0a-0cb1-423c-a122388f9ed5 [Follower]" leader= testing.go:149: server-e962818f-1d0a-0cb1-423c-a122388f9ed5: 2021-08-25T09:28:06.386Z [INFO] server-e962818f-1d0a-0cb1-423c-a122388f9ed5: entering follower state: follower="Node at e962818f-1d0a-0cb1-423c-a122388f9ed5 [Follower]" leader= testLoggerAdapter verbose: server-227f3f58-3e33-e7b6-2932-9026b2d38170: 2021-08-25T09:28:06.387Z [INFO] server-227f3f58-3e33-e7b6-2932-9026b2d38170: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27343939-c938-1f03-3869-4d341d79d44b Address:27343939-c938-1f03-3869-4d341d79d44b} {Suffrage:Voter ID:server-e962818f-1d0a-0cb1-423c-a122388f9ed5 Address:e962818f-1d0a-0cb1-423c-a122388f9ed5} {Suffrage:Voter ID:server-227f3f58-3e33-e7b6-2932-9026b2d38170 Address:227f3f58-3e33-e7b6-2932-9026b2d38170}]" testing.go:149: server-227f3f58-3e33-e7b6-2932-9026b2d38170: 2021-08-25T09:28:06.387Z [INFO] server-227f3f58-3e33-e7b6-2932-9026b2d38170: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27343939-c938-1f03-3869-4d341d79d44b Address:27343939-c938-1f03-3869-4d341d79d44b} {Suffrage:Voter ID:server-e962818f-1d0a-0cb1-423c-a122388f9ed5 Address:e962818f-1d0a-0cb1-423c-a122388f9ed5} {Suffrage:Voter ID:server-227f3f58-3e33-e7b6-2932-9026b2d38170 Address:227f3f58-3e33-e7b6-2932-9026b2d38170}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.387Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:06.387Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-227f3f58-3e33-e7b6-2932-9026b2d38170: 2021-08-25T09:28:06.387Z [INFO] server-227f3f58-3e33-e7b6-2932-9026b2d38170: entering follower state: follower="Node at 227f3f58-3e33-e7b6-2932-9026b2d38170 [Follower]" leader= testing.go:149: server-227f3f58-3e33-e7b6-2932-9026b2d38170: 2021-08-25T09:28:06.387Z [INFO] server-227f3f58-3e33-e7b6-2932-9026b2d38170: entering follower state: follower="Node at 227f3f58-3e33-e7b6-2932-9026b2d38170 [Follower]" leader= testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [WARN] server-27343939-c938-1f03-3869-4d341d79d44b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [WARN] server-27343939-c938-1f03-3869-4d341d79d44b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: entering candidate state: node="Node at 27343939-c938-1f03-3869-4d341d79d44b [Candidate]" term=2 testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: entering candidate state: node="Node at 27343939-c938-1f03-3869-4d341d79d44b [Candidate]" term=2 testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: election won: tally=2 testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: election won: tally=2 testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: entering leader state: leader="Node at 27343939-c938-1f03-3869-4d341d79d44b [Leader]" testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: entering leader state: leader="Node at 27343939-c938-1f03-3869-4d341d79d44b [Leader]" testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: added peer, starting replication: peer=server-e962818f-1d0a-0cb1-423c-a122388f9ed5 testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: added peer, starting replication: peer=server-e962818f-1d0a-0cb1-423c-a122388f9ed5 testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.440Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: added peer, starting replication: peer=server-227f3f58-3e33-e7b6-2932-9026b2d38170 testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.440Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: added peer, starting replication: peer=server-227f3f58-3e33-e7b6-2932-9026b2d38170 testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.440Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: pipelining replication: peer="{Voter server-227f3f58-3e33-e7b6-2932-9026b2d38170 227f3f58-3e33-e7b6-2932-9026b2d38170}" testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.440Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: pipelining replication: peer="{Voter server-227f3f58-3e33-e7b6-2932-9026b2d38170 227f3f58-3e33-e7b6-2932-9026b2d38170}" testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.440Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: pipelining replication: peer="{Voter server-e962818f-1d0a-0cb1-423c-a122388f9ed5 e962818f-1d0a-0cb1-423c-a122388f9ed5}" testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.440Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: pipelining replication: peer="{Voter server-e962818f-1d0a-0cb1-423c-a122388f9ed5 e962818f-1d0a-0cb1-423c-a122388f9ed5}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.545Z [INFO] stable state for Leader reached at 2021-08-25 09:28:06.440482781 +0000 UTC m=+6.204698710 (1 nodes), 53.282368ms from start of poll, 54.487336ms from cluster start. Timeout at 2021-08-25 09:28:06.545491182 +0000 UTC m=+6.309707131, 105.008421ms after stability testing.go:149: cluster: 2021-08-25T09:28:06.545Z [INFO] stable state for Leader reached at 2021-08-25 09:28:06.440482781 +0000 UTC m=+6.204698710 (1 nodes), 53.282368ms from start of poll, 54.487336ms from cluster start. Timeout at 2021-08-25 09:28:06.545491182 +0000 UTC m=+6.309707131, 105.008421ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.545Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:06.545Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.651Z [INFO] stable state for Follower reached at 2021-08-25 09:28:06.545841669 +0000 UTC m=+6.310057538 (2 nodes), 3.22µs from start of poll, 159.846164ms from cluster start. Timeout at 2021-08-25 09:28:06.650938901 +0000 UTC m=+6.415154870, 105.097332ms after stability testing.go:149: cluster: 2021-08-25T09:28:06.651Z [INFO] stable state for Follower reached at 2021-08-25 09:28:06.545841669 +0000 UTC m=+6.310057538 (2 nodes), 3.22µs from start of poll, 159.846164ms from cluster start. Timeout at 2021-08-25 09:28:06.650938901 +0000 UTC m=+6.415154870, 105.097332ms after stability testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.651Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: aborting pipeline replication: peer="{Voter server-227f3f58-3e33-e7b6-2932-9026b2d38170 227f3f58-3e33-e7b6-2932-9026b2d38170}" testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.651Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: aborting pipeline replication: peer="{Voter server-227f3f58-3e33-e7b6-2932-9026b2d38170 227f3f58-3e33-e7b6-2932-9026b2d38170}" testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.651Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: aborting pipeline replication: peer="{Voter server-e962818f-1d0a-0cb1-423c-a122388f9ed5 e962818f-1d0a-0cb1-423c-a122388f9ed5}" testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.651Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: aborting pipeline replication: peer="{Voter server-e962818f-1d0a-0cb1-423c-a122388f9ed5 e962818f-1d0a-0cb1-423c-a122388f9ed5}" testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.701Z [ERROR] server-27343939-c938-1f03-3869-4d341d79d44b: failed to heartbeat to: peer=e962818f-1d0a-0cb1-423c-a122388f9ed5 error="command timed out" testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.701Z [ERROR] server-27343939-c938-1f03-3869-4d341d79d44b: failed to heartbeat to: peer=e962818f-1d0a-0cb1-423c-a122388f9ed5 error="command timed out" --- PASS: TestRaft_ApplyNonLeader (0.32s) === RUN TestRaft_ApplyConcurrent testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.705Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b13f6cde-a725-f012-0112-f10901efd360 Address:b13f6cde-a725-f012-0112-f10901efd360} {Suffrage:Voter ID:server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 Address:7a1edcec-5dc4-5b3f-b39d-9f11a877f904} {Suffrage:Voter ID:server-bd21579c-c57b-59de-76c4-b761e79393ee Address:bd21579c-c57b-59de-76c4-b761e79393ee}]" testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.705Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b13f6cde-a725-f012-0112-f10901efd360 Address:b13f6cde-a725-f012-0112-f10901efd360} {Suffrage:Voter ID:server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 Address:7a1edcec-5dc4-5b3f-b39d-9f11a877f904} {Suffrage:Voter ID:server-bd21579c-c57b-59de-76c4-b761e79393ee Address:bd21579c-c57b-59de-76c4-b761e79393ee}]" testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.706Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: entering follower state: follower="Node at b13f6cde-a725-f012-0112-f10901efd360 [Follower]" leader= testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.706Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: entering follower state: follower="Node at b13f6cde-a725-f012-0112-f10901efd360 [Follower]" leader= testLoggerAdapter verbose: server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904: 2021-08-25T09:28:06.706Z [INFO] server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b13f6cde-a725-f012-0112-f10901efd360 Address:b13f6cde-a725-f012-0112-f10901efd360} {Suffrage:Voter ID:server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 Address:7a1edcec-5dc4-5b3f-b39d-9f11a877f904} {Suffrage:Voter ID:server-bd21579c-c57b-59de-76c4-b761e79393ee Address:bd21579c-c57b-59de-76c4-b761e79393ee}]" testing.go:149: server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904: 2021-08-25T09:28:06.706Z [INFO] server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b13f6cde-a725-f012-0112-f10901efd360 Address:b13f6cde-a725-f012-0112-f10901efd360} {Suffrage:Voter ID:server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 Address:7a1edcec-5dc4-5b3f-b39d-9f11a877f904} {Suffrage:Voter ID:server-bd21579c-c57b-59de-76c4-b761e79393ee Address:bd21579c-c57b-59de-76c4-b761e79393ee}]" testLoggerAdapter verbose: server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904: 2021-08-25T09:28:06.706Z [INFO] server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904: entering follower state: follower="Node at 7a1edcec-5dc4-5b3f-b39d-9f11a877f904 [Follower]" leader= testing.go:149: server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904: 2021-08-25T09:28:06.706Z [INFO] server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904: entering follower state: follower="Node at 7a1edcec-5dc4-5b3f-b39d-9f11a877f904 [Follower]" leader= testLoggerAdapter verbose: server-bd21579c-c57b-59de-76c4-b761e79393ee: 2021-08-25T09:28:06.706Z [INFO] server-bd21579c-c57b-59de-76c4-b761e79393ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b13f6cde-a725-f012-0112-f10901efd360 Address:b13f6cde-a725-f012-0112-f10901efd360} {Suffrage:Voter ID:server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 Address:7a1edcec-5dc4-5b3f-b39d-9f11a877f904} {Suffrage:Voter ID:server-bd21579c-c57b-59de-76c4-b761e79393ee Address:bd21579c-c57b-59de-76c4-b761e79393ee}]" testing.go:149: server-bd21579c-c57b-59de-76c4-b761e79393ee: 2021-08-25T09:28:06.706Z [INFO] server-bd21579c-c57b-59de-76c4-b761e79393ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b13f6cde-a725-f012-0112-f10901efd360 Address:b13f6cde-a725-f012-0112-f10901efd360} {Suffrage:Voter ID:server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 Address:7a1edcec-5dc4-5b3f-b39d-9f11a877f904} {Suffrage:Voter ID:server-bd21579c-c57b-59de-76c4-b761e79393ee Address:bd21579c-c57b-59de-76c4-b761e79393ee}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.706Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:06.706Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-bd21579c-c57b-59de-76c4-b761e79393ee: 2021-08-25T09:28:06.706Z [INFO] server-bd21579c-c57b-59de-76c4-b761e79393ee: entering follower state: follower="Node at bd21579c-c57b-59de-76c4-b761e79393ee [Follower]" leader= testing.go:149: server-bd21579c-c57b-59de-76c4-b761e79393ee: 2021-08-25T09:28:06.706Z [INFO] server-bd21579c-c57b-59de-76c4-b761e79393ee: entering follower state: follower="Node at bd21579c-c57b-59de-76c4-b761e79393ee [Follower]" leader= testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [WARN] server-b13f6cde-a725-f012-0112-f10901efd360: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [WARN] server-b13f6cde-a725-f012-0112-f10901efd360: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: entering candidate state: node="Node at b13f6cde-a725-f012-0112-f10901efd360 [Candidate]" term=2 testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: entering candidate state: node="Node at b13f6cde-a725-f012-0112-f10901efd360 [Candidate]" term=2 testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: election won: tally=2 testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: election won: tally=2 testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: entering leader state: leader="Node at b13f6cde-a725-f012-0112-f10901efd360 [Leader]" testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: entering leader state: leader="Node at b13f6cde-a725-f012-0112-f10901efd360 [Leader]" testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: added peer, starting replication: peer=server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: added peer, starting replication: peer=server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: added peer, starting replication: peer=server-bd21579c-c57b-59de-76c4-b761e79393ee testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: added peer, starting replication: peer=server-bd21579c-c57b-59de-76c4-b761e79393ee testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.818Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: pipelining replication: peer="{Voter server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 7a1edcec-5dc4-5b3f-b39d-9f11a877f904}" testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.818Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: pipelining replication: peer="{Voter server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 7a1edcec-5dc4-5b3f-b39d-9f11a877f904}" testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.818Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: pipelining replication: peer="{Voter server-bd21579c-c57b-59de-76c4-b761e79393ee bd21579c-c57b-59de-76c4-b761e79393ee}" testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.818Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: pipelining replication: peer="{Voter server-bd21579c-c57b-59de-76c4-b761e79393ee bd21579c-c57b-59de-76c4-b761e79393ee}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.023Z [INFO] stable state for Leader reached at 2021-08-25 09:28:06.817929921 +0000 UTC m=+6.582145830 (1 nodes), 111.049159ms from start of poll, 112.305662ms from cluster start. Timeout at 2021-08-25 09:28:07.023068355 +0000 UTC m=+6.787284264, 205.138434ms after stability testing.go:149: cluster: 2021-08-25T09:28:07.023Z [INFO] stable state for Leader reached at 2021-08-25 09:28:06.817929921 +0000 UTC m=+6.582145830 (1 nodes), 111.049159ms from start of poll, 112.305662ms from cluster start. Timeout at 2021-08-25 09:28:07.023068355 +0000 UTC m=+6.787284264, 205.138434ms after stability testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:07.035Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: aborting pipeline replication: peer="{Voter server-bd21579c-c57b-59de-76c4-b761e79393ee bd21579c-c57b-59de-76c4-b761e79393ee}" testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:07.035Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: aborting pipeline replication: peer="{Voter server-bd21579c-c57b-59de-76c4-b761e79393ee bd21579c-c57b-59de-76c4-b761e79393ee}" testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:07.035Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: aborting pipeline replication: peer="{Voter server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 7a1edcec-5dc4-5b3f-b39d-9f11a877f904}" testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:07.035Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: aborting pipeline replication: peer="{Voter server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 7a1edcec-5dc4-5b3f-b39d-9f11a877f904}" --- PASS: TestRaft_ApplyConcurrent (0.33s) === RUN TestRaft_ApplyConcurrent_Timeout testLoggerAdapter verbose: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.038Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64230c52-0049-cf61-f6b3-03d525a099bd Address:64230c52-0049-cf61-f6b3-03d525a099bd}]" testing.go:149: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.038Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64230c52-0049-cf61-f6b3-03d525a099bd Address:64230c52-0049-cf61-f6b3-03d525a099bd}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.038Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:07.038Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.038Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: entering follower state: follower="Node at 64230c52-0049-cf61-f6b3-03d525a099bd [Follower]" leader= testing.go:149: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.038Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: entering follower state: follower="Node at 64230c52-0049-cf61-f6b3-03d525a099bd [Follower]" leader= testLoggerAdapter verbose: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.230Z [WARN] server-64230c52-0049-cf61-f6b3-03d525a099bd: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.230Z [WARN] server-64230c52-0049-cf61-f6b3-03d525a099bd: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.230Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: entering candidate state: node="Node at 64230c52-0049-cf61-f6b3-03d525a099bd [Candidate]" term=2 testing.go:149: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.230Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: entering candidate state: node="Node at 64230c52-0049-cf61-f6b3-03d525a099bd [Candidate]" term=2 testLoggerAdapter verbose: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.230Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: election won: tally=1 testing.go:149: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.230Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: election won: tally=1 testLoggerAdapter verbose: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.230Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: entering leader state: leader="Node at 64230c52-0049-cf61-f6b3-03d525a099bd [Leader]" testing.go:149: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.230Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: entering leader state: leader="Node at 64230c52-0049-cf61-f6b3-03d525a099bd [Leader]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.431Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.230685219 +0000 UTC m=+6.994901128 (1 nodes), 191.980557ms from start of poll, 192.369661ms from cluster start. Timeout at 2021-08-25 09:28:07.431798048 +0000 UTC m=+7.196014017, 201.112889ms after stability testing.go:149: cluster: 2021-08-25T09:28:07.431Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.230685219 +0000 UTC m=+6.994901128 (1 nodes), 191.980557ms from start of poll, 192.369661ms from cluster start. Timeout at 2021-08-25 09:28:07.431798048 +0000 UTC m=+7.196014017, 201.112889ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.40s) === RUN TestRaft_JoinNode testLoggerAdapter verbose: server-7123646f-1f46-adbf-b3d0-b3f4673ff95f: 2021-08-25T09:28:07.434Z [INFO] server-7123646f-1f46-adbf-b3d0-b3f4673ff95f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7123646f-1f46-adbf-b3d0-b3f4673ff95f Address:7123646f-1f46-adbf-b3d0-b3f4673ff95f} {Suffrage:Voter ID:server-4d015a90-64bd-4573-253e-3dbecc1e887a Address:4d015a90-64bd-4573-253e-3dbecc1e887a}]" testing.go:149: server-7123646f-1f46-adbf-b3d0-b3f4673ff95f: 2021-08-25T09:28:07.434Z [INFO] server-7123646f-1f46-adbf-b3d0-b3f4673ff95f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7123646f-1f46-adbf-b3d0-b3f4673ff95f Address:7123646f-1f46-adbf-b3d0-b3f4673ff95f} {Suffrage:Voter ID:server-4d015a90-64bd-4573-253e-3dbecc1e887a Address:4d015a90-64bd-4573-253e-3dbecc1e887a}]" testLoggerAdapter verbose: server-7123646f-1f46-adbf-b3d0-b3f4673ff95f: 2021-08-25T09:28:07.434Z [INFO] server-7123646f-1f46-adbf-b3d0-b3f4673ff95f: entering follower state: follower="Node at 7123646f-1f46-adbf-b3d0-b3f4673ff95f [Follower]" leader= testing.go:149: server-7123646f-1f46-adbf-b3d0-b3f4673ff95f: 2021-08-25T09:28:07.434Z [INFO] server-7123646f-1f46-adbf-b3d0-b3f4673ff95f: entering follower state: follower="Node at 7123646f-1f46-adbf-b3d0-b3f4673ff95f [Follower]" leader= testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.434Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7123646f-1f46-adbf-b3d0-b3f4673ff95f Address:7123646f-1f46-adbf-b3d0-b3f4673ff95f} {Suffrage:Voter ID:server-4d015a90-64bd-4573-253e-3dbecc1e887a Address:4d015a90-64bd-4573-253e-3dbecc1e887a}]" testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.434Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7123646f-1f46-adbf-b3d0-b3f4673ff95f Address:7123646f-1f46-adbf-b3d0-b3f4673ff95f} {Suffrage:Voter ID:server-4d015a90-64bd-4573-253e-3dbecc1e887a Address:4d015a90-64bd-4573-253e-3dbecc1e887a}]" testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.434Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: entering follower state: follower="Node at 4d015a90-64bd-4573-253e-3dbecc1e887a [Follower]" leader= testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.434Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: entering follower state: follower="Node at 4d015a90-64bd-4573-253e-3dbecc1e887a [Follower]" leader= testLoggerAdapter verbose: server-6f6c6b55-66a9-f17c-bb21-87481a40843f: 2021-08-25T09:28:07.435Z [INFO] server-6f6c6b55-66a9-f17c-bb21-87481a40843f: initial configuration: index=0 servers=[] testing.go:149: server-6f6c6b55-66a9-f17c-bb21-87481a40843f: 2021-08-25T09:28:07.435Z [INFO] server-6f6c6b55-66a9-f17c-bb21-87481a40843f: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.435Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:07.435Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-6f6c6b55-66a9-f17c-bb21-87481a40843f: 2021-08-25T09:28:07.435Z [INFO] server-6f6c6b55-66a9-f17c-bb21-87481a40843f: entering follower state: follower="Node at 6f6c6b55-66a9-f17c-bb21-87481a40843f [Follower]" leader= testing.go:149: server-6f6c6b55-66a9-f17c-bb21-87481a40843f: 2021-08-25T09:28:07.435Z [INFO] server-6f6c6b55-66a9-f17c-bb21-87481a40843f: entering follower state: follower="Node at 6f6c6b55-66a9-f17c-bb21-87481a40843f [Follower]" leader= testLoggerAdapter verbose: server-6f6c6b55-66a9-f17c-bb21-87481a40843f: 2021-08-25T09:28:07.488Z [WARN] server-6f6c6b55-66a9-f17c-bb21-87481a40843f: no known peers, aborting election testing.go:149: server-6f6c6b55-66a9-f17c-bb21-87481a40843f: 2021-08-25T09:28:07.488Z [WARN] server-6f6c6b55-66a9-f17c-bb21-87481a40843f: no known peers, aborting election testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.491Z [WARN] server-4d015a90-64bd-4573-253e-3dbecc1e887a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.491Z [WARN] server-4d015a90-64bd-4573-253e-3dbecc1e887a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.491Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: entering candidate state: node="Node at 4d015a90-64bd-4573-253e-3dbecc1e887a [Candidate]" term=2 testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.491Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: entering candidate state: node="Node at 4d015a90-64bd-4573-253e-3dbecc1e887a [Candidate]" term=2 testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.492Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: election won: tally=2 testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.492Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: election won: tally=2 testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.492Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: entering leader state: leader="Node at 4d015a90-64bd-4573-253e-3dbecc1e887a [Leader]" testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.492Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: entering leader state: leader="Node at 4d015a90-64bd-4573-253e-3dbecc1e887a [Leader]" testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.492Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: added peer, starting replication: peer=server-7123646f-1f46-adbf-b3d0-b3f4673ff95f testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.492Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: added peer, starting replication: peer=server-7123646f-1f46-adbf-b3d0-b3f4673ff95f testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.492Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: pipelining replication: peer="{Voter server-7123646f-1f46-adbf-b3d0-b3f4673ff95f 7123646f-1f46-adbf-b3d0-b3f4673ff95f}" testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.492Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: pipelining replication: peer="{Voter server-7123646f-1f46-adbf-b3d0-b3f4673ff95f 7123646f-1f46-adbf-b3d0-b3f4673ff95f}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.597Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.492304164 +0000 UTC m=+7.256520073 (1 nodes), 56.676552ms from start of poll, 58.048465ms from cluster start. Timeout at 2021-08-25 09:28:07.597487889 +0000 UTC m=+7.361703798, 105.183725ms after stability testing.go:149: cluster: 2021-08-25T09:28:07.597Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.492304164 +0000 UTC m=+7.256520073 (1 nodes), 56.676552ms from start of poll, 58.048465ms from cluster start. Timeout at 2021-08-25 09:28:07.597487889 +0000 UTC m=+7.361703798, 105.183725ms after stability testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.597Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: updating configuration: command=AddStaging server-id=server-6f6c6b55-66a9-f17c-bb21-87481a40843f server-addr=6f6c6b55-66a9-f17c-bb21-87481a40843f servers="[{Suffrage:Voter ID:server-7123646f-1f46-adbf-b3d0-b3f4673ff95f Address:7123646f-1f46-adbf-b3d0-b3f4673ff95f} {Suffrage:Voter ID:server-4d015a90-64bd-4573-253e-3dbecc1e887a Address:4d015a90-64bd-4573-253e-3dbecc1e887a} {Suffrage:Voter ID:server-6f6c6b55-66a9-f17c-bb21-87481a40843f Address:6f6c6b55-66a9-f17c-bb21-87481a40843f}]" testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.597Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: updating configuration: command=AddStaging server-id=server-6f6c6b55-66a9-f17c-bb21-87481a40843f server-addr=6f6c6b55-66a9-f17c-bb21-87481a40843f servers="[{Suffrage:Voter ID:server-7123646f-1f46-adbf-b3d0-b3f4673ff95f Address:7123646f-1f46-adbf-b3d0-b3f4673ff95f} {Suffrage:Voter ID:server-4d015a90-64bd-4573-253e-3dbecc1e887a Address:4d015a90-64bd-4573-253e-3dbecc1e887a} {Suffrage:Voter ID:server-6f6c6b55-66a9-f17c-bb21-87481a40843f Address:6f6c6b55-66a9-f17c-bb21-87481a40843f}]" testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.597Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: added peer, starting replication: peer=server-6f6c6b55-66a9-f17c-bb21-87481a40843f testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.597Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: added peer, starting replication: peer=server-6f6c6b55-66a9-f17c-bb21-87481a40843f testLoggerAdapter verbose: server-6f6c6b55-66a9-f17c-bb21-87481a40843f: 2021-08-25T09:28:07.598Z [WARN] server-6f6c6b55-66a9-f17c-bb21-87481a40843f: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-6f6c6b55-66a9-f17c-bb21-87481a40843f: 2021-08-25T09:28:07.598Z [WARN] server-6f6c6b55-66a9-f17c-bb21-87481a40843f: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.598Z [WARN] server-4d015a90-64bd-4573-253e-3dbecc1e887a: appendEntries rejected, sending older logs: peer="{Voter server-6f6c6b55-66a9-f17c-bb21-87481a40843f 6f6c6b55-66a9-f17c-bb21-87481a40843f}" next=1 testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.598Z [WARN] server-4d015a90-64bd-4573-253e-3dbecc1e887a: appendEntries rejected, sending older logs: peer="{Voter server-6f6c6b55-66a9-f17c-bb21-87481a40843f 6f6c6b55-66a9-f17c-bb21-87481a40843f}" next=1 testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.598Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:07.598Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.598Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: pipelining replication: peer="{Voter server-6f6c6b55-66a9-f17c-bb21-87481a40843f 6f6c6b55-66a9-f17c-bb21-87481a40843f}" testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.598Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: pipelining replication: peer="{Voter server-6f6c6b55-66a9-f17c-bb21-87481a40843f 6f6c6b55-66a9-f17c-bb21-87481a40843f}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.703Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.598341009 +0000 UTC m=+7.362556898 (1 nodes), 2.099µs from start of poll, 164.08529ms from cluster start. Timeout at 2021-08-25 09:28:07.703461939 +0000 UTC m=+7.467677848, 105.12095ms after stability testing.go:149: cluster: 2021-08-25T09:28:07.703Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.598341009 +0000 UTC m=+7.362556898 (1 nodes), 2.099µs from start of poll, 164.08529ms from cluster start. Timeout at 2021-08-25 09:28:07.703461939 +0000 UTC m=+7.467677848, 105.12095ms after stability testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.703Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: aborting pipeline replication: peer="{Voter server-7123646f-1f46-adbf-b3d0-b3f4673ff95f 7123646f-1f46-adbf-b3d0-b3f4673ff95f}" testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.703Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: aborting pipeline replication: peer="{Voter server-7123646f-1f46-adbf-b3d0-b3f4673ff95f 7123646f-1f46-adbf-b3d0-b3f4673ff95f}" testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.703Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: aborting pipeline replication: peer="{Voter server-6f6c6b55-66a9-f17c-bb21-87481a40843f 6f6c6b55-66a9-f17c-bb21-87481a40843f}" testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.703Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: aborting pipeline replication: peer="{Voter server-6f6c6b55-66a9-f17c-bb21-87481a40843f 6f6c6b55-66a9-f17c-bb21-87481a40843f}" --- PASS: TestRaft_JoinNode (0.27s) === RUN TestRaft_JoinNode_ConfigStore testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.706Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9 Address:09f20c95-f8df-5131-f93f-3cd5e1bfcba9}]" testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.706Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9 Address:09f20c95-f8df-5131-f93f-3cd5e1bfcba9}]" testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.706Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: entering follower state: follower="Node at 09f20c95-f8df-5131-f93f-3cd5e1bfcba9 [Follower]" leader= testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.706Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: entering follower state: follower="Node at 09f20c95-f8df-5131-f93f-3cd5e1bfcba9 [Follower]" leader= testLoggerAdapter verbose: server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: 2021-08-25T09:28:07.707Z [INFO] server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: initial configuration: index=0 servers=[] testing.go:149: server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: 2021-08-25T09:28:07.707Z [INFO] server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: 2021-08-25T09:28:07.707Z [INFO] server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: entering follower state: follower="Node at c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c [Follower]" leader= testing.go:149: server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: 2021-08-25T09:28:07.707Z [INFO] server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: entering follower state: follower="Node at c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c [Follower]" leader= testLoggerAdapter verbose: server-774530ec-2a41-28cd-d722-da8f2b52b59b: 2021-08-25T09:28:07.707Z [INFO] server-774530ec-2a41-28cd-d722-da8f2b52b59b: initial configuration: index=0 servers=[] testing.go:149: server-774530ec-2a41-28cd-d722-da8f2b52b59b: 2021-08-25T09:28:07.707Z [INFO] server-774530ec-2a41-28cd-d722-da8f2b52b59b: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.707Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:07.707Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-774530ec-2a41-28cd-d722-da8f2b52b59b: 2021-08-25T09:28:07.707Z [INFO] server-774530ec-2a41-28cd-d722-da8f2b52b59b: entering follower state: follower="Node at 774530ec-2a41-28cd-d722-da8f2b52b59b [Follower]" leader= testing.go:149: server-774530ec-2a41-28cd-d722-da8f2b52b59b: 2021-08-25T09:28:07.707Z [INFO] server-774530ec-2a41-28cd-d722-da8f2b52b59b: entering follower state: follower="Node at 774530ec-2a41-28cd-d722-da8f2b52b59b [Follower]" leader= testLoggerAdapter verbose: server-774530ec-2a41-28cd-d722-da8f2b52b59b: 2021-08-25T09:28:07.787Z [WARN] server-774530ec-2a41-28cd-d722-da8f2b52b59b: no known peers, aborting election testing.go:149: server-774530ec-2a41-28cd-d722-da8f2b52b59b: 2021-08-25T09:28:07.787Z [WARN] server-774530ec-2a41-28cd-d722-da8f2b52b59b: no known peers, aborting election testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.789Z [WARN] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.789Z [WARN] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.790Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: entering candidate state: node="Node at 09f20c95-f8df-5131-f93f-3cd5e1bfcba9 [Candidate]" term=2 testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.790Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: entering candidate state: node="Node at 09f20c95-f8df-5131-f93f-3cd5e1bfcba9 [Candidate]" term=2 testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.790Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: election won: tally=1 testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.790Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: election won: tally=1 testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.790Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: entering leader state: leader="Node at 09f20c95-f8df-5131-f93f-3cd5e1bfcba9 [Leader]" testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.790Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: entering leader state: leader="Node at 09f20c95-f8df-5131-f93f-3cd5e1bfcba9 [Leader]" testLoggerAdapter verbose: server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: 2021-08-25T09:28:07.795Z [WARN] server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: no known peers, aborting election testing.go:149: server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: 2021-08-25T09:28:07.795Z [WARN] server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: no known peers, aborting election testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.895Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.790453134 +0000 UTC m=+7.554669023 (1 nodes), 82.593403ms from start of poll, 84.289366ms from cluster start. Timeout at 2021-08-25 09:28:07.895609621 +0000 UTC m=+7.659825570, 105.156547ms after stability testing.go:149: cluster: 2021-08-25T09:28:07.895Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.790453134 +0000 UTC m=+7.554669023 (1 nodes), 82.593403ms from start of poll, 84.289366ms from cluster start. Timeout at 2021-08-25 09:28:07.895609621 +0000 UTC m=+7.659825570, 105.156547ms after stability testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.895Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: updating configuration: command=AddStaging server-id=server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c server-addr=c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c servers="[{Suffrage:Voter ID:server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9 Address:09f20c95-f8df-5131-f93f-3cd5e1bfcba9} {Suffrage:Voter ID:server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c Address:c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c}]" testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.895Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: updating configuration: command=AddStaging server-id=server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c server-addr=c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c servers="[{Suffrage:Voter ID:server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9 Address:09f20c95-f8df-5131-f93f-3cd5e1bfcba9} {Suffrage:Voter ID:server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c Address:c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c}]" testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.896Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: added peer, starting replication: peer=server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.896Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: added peer, starting replication: peer=server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.896Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:07.896Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: 2021-08-25T09:28:07.896Z [WARN] server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: 2021-08-25T09:28:07.896Z [WARN] server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.896Z [WARN] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: appendEntries rejected, sending older logs: peer="{Voter server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c}" next=1 testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.896Z [WARN] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: appendEntries rejected, sending older logs: peer="{Voter server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c}" next=1 testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.896Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: pipelining replication: peer="{Voter server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c}" testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.896Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: pipelining replication: peer="{Voter server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.001Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.896359911 +0000 UTC m=+7.660575800 (1 nodes), 1.859µs from start of poll, 190.196143ms from cluster start. Timeout at 2021-08-25 09:28:08.001450124 +0000 UTC m=+7.765666073, 105.090273ms after stability testing.go:149: cluster: 2021-08-25T09:28:08.001Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.896359911 +0000 UTC m=+7.660575800 (1 nodes), 1.859µs from start of poll, 190.196143ms from cluster start. Timeout at 2021-08-25 09:28:08.001450124 +0000 UTC m=+7.765666073, 105.090273ms after stability testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.001Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: updating configuration: command=AddStaging server-id=server-774530ec-2a41-28cd-d722-da8f2b52b59b server-addr=774530ec-2a41-28cd-d722-da8f2b52b59b servers="[{Suffrage:Voter ID:server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9 Address:09f20c95-f8df-5131-f93f-3cd5e1bfcba9} {Suffrage:Voter ID:server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c Address:c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c} {Suffrage:Voter ID:server-774530ec-2a41-28cd-d722-da8f2b52b59b Address:774530ec-2a41-28cd-d722-da8f2b52b59b}]" testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.001Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: updating configuration: command=AddStaging server-id=server-774530ec-2a41-28cd-d722-da8f2b52b59b server-addr=774530ec-2a41-28cd-d722-da8f2b52b59b servers="[{Suffrage:Voter ID:server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9 Address:09f20c95-f8df-5131-f93f-3cd5e1bfcba9} {Suffrage:Voter ID:server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c Address:c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c} {Suffrage:Voter ID:server-774530ec-2a41-28cd-d722-da8f2b52b59b Address:774530ec-2a41-28cd-d722-da8f2b52b59b}]" testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.001Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: added peer, starting replication: peer=server-774530ec-2a41-28cd-d722-da8f2b52b59b testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.001Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: added peer, starting replication: peer=server-774530ec-2a41-28cd-d722-da8f2b52b59b testLoggerAdapter verbose: server-774530ec-2a41-28cd-d722-da8f2b52b59b: 2021-08-25T09:28:08.002Z [WARN] server-774530ec-2a41-28cd-d722-da8f2b52b59b: failed to get previous log: previous-index=4 last-index=0 error="log not found" testing.go:149: server-774530ec-2a41-28cd-d722-da8f2b52b59b: 2021-08-25T09:28:08.002Z [WARN] server-774530ec-2a41-28cd-d722-da8f2b52b59b: failed to get previous log: previous-index=4 last-index=0 error="log not found" testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.002Z [WARN] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: appendEntries rejected, sending older logs: peer="{Voter server-774530ec-2a41-28cd-d722-da8f2b52b59b 774530ec-2a41-28cd-d722-da8f2b52b59b}" next=1 testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.002Z [WARN] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: appendEntries rejected, sending older logs: peer="{Voter server-774530ec-2a41-28cd-d722-da8f2b52b59b 774530ec-2a41-28cd-d722-da8f2b52b59b}" next=1 testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.002Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:08.002Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.002Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: pipelining replication: peer="{Voter server-774530ec-2a41-28cd-d722-da8f2b52b59b 774530ec-2a41-28cd-d722-da8f2b52b59b}" testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.002Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: pipelining replication: peer="{Voter server-774530ec-2a41-28cd-d722-da8f2b52b59b 774530ec-2a41-28cd-d722-da8f2b52b59b}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.107Z [INFO] stable state for Leader reached at 2021-08-25 09:28:08.002346421 +0000 UTC m=+7.766562310 (1 nodes), 1.94µs from start of poll, 296.182653ms from cluster start. Timeout at 2021-08-25 09:28:08.107428395 +0000 UTC m=+7.871644304, 105.081994ms after stability testing.go:149: cluster: 2021-08-25T09:28:08.107Z [INFO] stable state for Leader reached at 2021-08-25 09:28:08.002346421 +0000 UTC m=+7.766562310 (1 nodes), 1.94µs from start of poll, 296.182653ms from cluster start. Timeout at 2021-08-25 09:28:08.107428395 +0000 UTC m=+7.871644304, 105.081994ms after stability testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.107Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: aborting pipeline replication: peer="{Voter server-774530ec-2a41-28cd-d722-da8f2b52b59b 774530ec-2a41-28cd-d722-da8f2b52b59b}" testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.107Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: aborting pipeline replication: peer="{Voter server-774530ec-2a41-28cd-d722-da8f2b52b59b 774530ec-2a41-28cd-d722-da8f2b52b59b}" testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.108Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: aborting pipeline replication: peer="{Voter server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c}" testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.108Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: aborting pipeline replication: peer="{Voter server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c}" --- PASS: TestRaft_JoinNode_ConfigStore (0.41s) === RUN TestRaft_RemoveFollower testLoggerAdapter verbose: server-9b89ba74-8aee-e236-8eee-b67204f5351a: 2021-08-25T09:28:08.113Z [INFO] server-9b89ba74-8aee-e236-8eee-b67204f5351a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b89ba74-8aee-e236-8eee-b67204f5351a Address:9b89ba74-8aee-e236-8eee-b67204f5351a} {Suffrage:Voter ID:server-dd3282c0-b14d-5efa-0e74-3a06ac823338 Address:dd3282c0-b14d-5efa-0e74-3a06ac823338} {Suffrage:Voter ID:server-be5e9eef-f4ac-cc10-3b82-83df05e62605 Address:be5e9eef-f4ac-cc10-3b82-83df05e62605}]" testing.go:149: server-9b89ba74-8aee-e236-8eee-b67204f5351a: 2021-08-25T09:28:08.113Z [INFO] server-9b89ba74-8aee-e236-8eee-b67204f5351a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b89ba74-8aee-e236-8eee-b67204f5351a Address:9b89ba74-8aee-e236-8eee-b67204f5351a} {Suffrage:Voter ID:server-dd3282c0-b14d-5efa-0e74-3a06ac823338 Address:dd3282c0-b14d-5efa-0e74-3a06ac823338} {Suffrage:Voter ID:server-be5e9eef-f4ac-cc10-3b82-83df05e62605 Address:be5e9eef-f4ac-cc10-3b82-83df05e62605}]" testLoggerAdapter verbose: server-9b89ba74-8aee-e236-8eee-b67204f5351a: 2021-08-25T09:28:08.113Z [INFO] server-9b89ba74-8aee-e236-8eee-b67204f5351a: entering follower state: follower="Node at 9b89ba74-8aee-e236-8eee-b67204f5351a [Follower]" leader= testing.go:149: server-9b89ba74-8aee-e236-8eee-b67204f5351a: 2021-08-25T09:28:08.113Z [INFO] server-9b89ba74-8aee-e236-8eee-b67204f5351a: entering follower state: follower="Node at 9b89ba74-8aee-e236-8eee-b67204f5351a [Follower]" leader= testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.113Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b89ba74-8aee-e236-8eee-b67204f5351a Address:9b89ba74-8aee-e236-8eee-b67204f5351a} {Suffrage:Voter ID:server-dd3282c0-b14d-5efa-0e74-3a06ac823338 Address:dd3282c0-b14d-5efa-0e74-3a06ac823338} {Suffrage:Voter ID:server-be5e9eef-f4ac-cc10-3b82-83df05e62605 Address:be5e9eef-f4ac-cc10-3b82-83df05e62605}]" testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.113Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b89ba74-8aee-e236-8eee-b67204f5351a Address:9b89ba74-8aee-e236-8eee-b67204f5351a} {Suffrage:Voter ID:server-dd3282c0-b14d-5efa-0e74-3a06ac823338 Address:dd3282c0-b14d-5efa-0e74-3a06ac823338} {Suffrage:Voter ID:server-be5e9eef-f4ac-cc10-3b82-83df05e62605 Address:be5e9eef-f4ac-cc10-3b82-83df05e62605}]" testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.113Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: entering follower state: follower="Node at dd3282c0-b14d-5efa-0e74-3a06ac823338 [Follower]" leader= testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.113Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: entering follower state: follower="Node at dd3282c0-b14d-5efa-0e74-3a06ac823338 [Follower]" leader= testLoggerAdapter verbose: server-be5e9eef-f4ac-cc10-3b82-83df05e62605: 2021-08-25T09:28:08.114Z [INFO] server-be5e9eef-f4ac-cc10-3b82-83df05e62605: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b89ba74-8aee-e236-8eee-b67204f5351a Address:9b89ba74-8aee-e236-8eee-b67204f5351a} {Suffrage:Voter ID:server-dd3282c0-b14d-5efa-0e74-3a06ac823338 Address:dd3282c0-b14d-5efa-0e74-3a06ac823338} {Suffrage:Voter ID:server-be5e9eef-f4ac-cc10-3b82-83df05e62605 Address:be5e9eef-f4ac-cc10-3b82-83df05e62605}]" testing.go:149: server-be5e9eef-f4ac-cc10-3b82-83df05e62605: 2021-08-25T09:28:08.114Z [INFO] server-be5e9eef-f4ac-cc10-3b82-83df05e62605: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b89ba74-8aee-e236-8eee-b67204f5351a Address:9b89ba74-8aee-e236-8eee-b67204f5351a} {Suffrage:Voter ID:server-dd3282c0-b14d-5efa-0e74-3a06ac823338 Address:dd3282c0-b14d-5efa-0e74-3a06ac823338} {Suffrage:Voter ID:server-be5e9eef-f4ac-cc10-3b82-83df05e62605 Address:be5e9eef-f4ac-cc10-3b82-83df05e62605}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.114Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:08.114Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-be5e9eef-f4ac-cc10-3b82-83df05e62605: 2021-08-25T09:28:08.114Z [INFO] server-be5e9eef-f4ac-cc10-3b82-83df05e62605: entering follower state: follower="Node at be5e9eef-f4ac-cc10-3b82-83df05e62605 [Follower]" leader= testing.go:149: server-be5e9eef-f4ac-cc10-3b82-83df05e62605: 2021-08-25T09:28:08.114Z [INFO] server-be5e9eef-f4ac-cc10-3b82-83df05e62605: entering follower state: follower="Node at be5e9eef-f4ac-cc10-3b82-83df05e62605 [Follower]" leader= testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [WARN] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [WARN] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: entering candidate state: node="Node at dd3282c0-b14d-5efa-0e74-3a06ac823338 [Candidate]" term=2 testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: entering candidate state: node="Node at dd3282c0-b14d-5efa-0e74-3a06ac823338 [Candidate]" term=2 testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: election won: tally=2 testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: election won: tally=2 testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: entering leader state: leader="Node at dd3282c0-b14d-5efa-0e74-3a06ac823338 [Leader]" testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: entering leader state: leader="Node at dd3282c0-b14d-5efa-0e74-3a06ac823338 [Leader]" testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: added peer, starting replication: peer=server-9b89ba74-8aee-e236-8eee-b67204f5351a testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: added peer, starting replication: peer=server-9b89ba74-8aee-e236-8eee-b67204f5351a testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: added peer, starting replication: peer=server-be5e9eef-f4ac-cc10-3b82-83df05e62605 testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: added peer, starting replication: peer=server-be5e9eef-f4ac-cc10-3b82-83df05e62605 testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: pipelining replication: peer="{Voter server-9b89ba74-8aee-e236-8eee-b67204f5351a 9b89ba74-8aee-e236-8eee-b67204f5351a}" testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: pipelining replication: peer="{Voter server-9b89ba74-8aee-e236-8eee-b67204f5351a 9b89ba74-8aee-e236-8eee-b67204f5351a}" testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.184Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: pipelining replication: peer="{Voter server-be5e9eef-f4ac-cc10-3b82-83df05e62605 be5e9eef-f4ac-cc10-3b82-83df05e62605}" testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.184Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: pipelining replication: peer="{Voter server-be5e9eef-f4ac-cc10-3b82-83df05e62605 be5e9eef-f4ac-cc10-3b82-83df05e62605}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.288Z [INFO] stable state for Leader reached at 2021-08-25 09:28:08.183739862 +0000 UTC m=+7.947955771 (1 nodes), 69.447545ms from start of poll, 70.613077ms from cluster start. Timeout at 2021-08-25 09:28:08.288864912 +0000 UTC m=+8.053080841, 105.12507ms after stability testing.go:149: cluster: 2021-08-25T09:28:08.288Z [INFO] stable state for Leader reached at 2021-08-25 09:28:08.183739862 +0000 UTC m=+7.947955771 (1 nodes), 69.447545ms from start of poll, 70.613077ms from cluster start. Timeout at 2021-08-25 09:28:08.288864912 +0000 UTC m=+8.053080841, 105.12507ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.294Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:08.294Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.399Z [INFO] stable state for Follower reached at 2021-08-25 09:28:08.294394998 +0000 UTC m=+8.058610907 (2 nodes), 2.98µs from start of poll, 181.268213ms from cluster start. Timeout at 2021-08-25 09:28:08.399516308 +0000 UTC m=+8.163732217, 105.12131ms after stability testing.go:149: cluster: 2021-08-25T09:28:08.399Z [INFO] stable state for Follower reached at 2021-08-25 09:28:08.294394998 +0000 UTC m=+8.058610907 (2 nodes), 2.98µs from start of poll, 181.268213ms from cluster start. Timeout at 2021-08-25 09:28:08.399516308 +0000 UTC m=+8.163732217, 105.12131ms after stability testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.399Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: updating configuration: command=RemoveServer server-id=server-9b89ba74-8aee-e236-8eee-b67204f5351a server-addr= servers="[{Suffrage:Voter ID:server-dd3282c0-b14d-5efa-0e74-3a06ac823338 Address:dd3282c0-b14d-5efa-0e74-3a06ac823338} {Suffrage:Voter ID:server-be5e9eef-f4ac-cc10-3b82-83df05e62605 Address:be5e9eef-f4ac-cc10-3b82-83df05e62605}]" testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.399Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: updating configuration: command=RemoveServer server-id=server-9b89ba74-8aee-e236-8eee-b67204f5351a server-addr= servers="[{Suffrage:Voter ID:server-dd3282c0-b14d-5efa-0e74-3a06ac823338 Address:dd3282c0-b14d-5efa-0e74-3a06ac823338} {Suffrage:Voter ID:server-be5e9eef-f4ac-cc10-3b82-83df05e62605 Address:be5e9eef-f4ac-cc10-3b82-83df05e62605}]" testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.400Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: removed peer, stopping replication: peer=server-9b89ba74-8aee-e236-8eee-b67204f5351a last-index=3 testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.400Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: removed peer, stopping replication: peer=server-9b89ba74-8aee-e236-8eee-b67204f5351a last-index=3 testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.400Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: aborting pipeline replication: peer="{Voter server-9b89ba74-8aee-e236-8eee-b67204f5351a 9b89ba74-8aee-e236-8eee-b67204f5351a}" testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.400Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: aborting pipeline replication: peer="{Voter server-9b89ba74-8aee-e236-8eee-b67204f5351a 9b89ba74-8aee-e236-8eee-b67204f5351a}" testLoggerAdapter verbose: server-9b89ba74-8aee-e236-8eee-b67204f5351a: 2021-08-25T09:28:08.461Z [WARN] server-9b89ba74-8aee-e236-8eee-b67204f5351a: heartbeat timeout reached, starting election: last-leader=dd3282c0-b14d-5efa-0e74-3a06ac823338 testing.go:149: server-9b89ba74-8aee-e236-8eee-b67204f5351a: 2021-08-25T09:28:08.461Z [WARN] server-9b89ba74-8aee-e236-8eee-b67204f5351a: heartbeat timeout reached, starting election: last-leader=dd3282c0-b14d-5efa-0e74-3a06ac823338 testLoggerAdapter verbose: server-9b89ba74-8aee-e236-8eee-b67204f5351a: 2021-08-25T09:28:08.461Z [INFO] server-9b89ba74-8aee-e236-8eee-b67204f5351a: entering candidate state: node="Node at 9b89ba74-8aee-e236-8eee-b67204f5351a [Candidate]" term=3 testing.go:149: server-9b89ba74-8aee-e236-8eee-b67204f5351a: 2021-08-25T09:28:08.461Z [INFO] server-9b89ba74-8aee-e236-8eee-b67204f5351a: entering candidate state: node="Node at 9b89ba74-8aee-e236-8eee-b67204f5351a [Candidate]" term=3 testLoggerAdapter verbose: server-be5e9eef-f4ac-cc10-3b82-83df05e62605: 2021-08-25T09:28:08.461Z [WARN] server-be5e9eef-f4ac-cc10-3b82-83df05e62605: rejecting vote request since we have a leader: from=9b89ba74-8aee-e236-8eee-b67204f5351a leader=dd3282c0-b14d-5efa-0e74-3a06ac823338 testing.go:149: server-be5e9eef-f4ac-cc10-3b82-83df05e62605: 2021-08-25T09:28:08.461Z [WARN] server-be5e9eef-f4ac-cc10-3b82-83df05e62605: rejecting vote request since we have a leader: from=9b89ba74-8aee-e236-8eee-b67204f5351a leader=dd3282c0-b14d-5efa-0e74-3a06ac823338 testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.461Z [WARN] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: rejecting vote request since we have a leader: from=9b89ba74-8aee-e236-8eee-b67204f5351a leader=dd3282c0-b14d-5efa-0e74-3a06ac823338 testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.461Z [WARN] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: rejecting vote request since we have a leader: from=9b89ba74-8aee-e236-8eee-b67204f5351a leader=dd3282c0-b14d-5efa-0e74-3a06ac823338 testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.505Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: aborting pipeline replication: peer="{Voter server-be5e9eef-f4ac-cc10-3b82-83df05e62605 be5e9eef-f4ac-cc10-3b82-83df05e62605}" testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.505Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: aborting pipeline replication: peer="{Voter server-be5e9eef-f4ac-cc10-3b82-83df05e62605 be5e9eef-f4ac-cc10-3b82-83df05e62605}" --- PASS: TestRaft_RemoveFollower (0.40s) === RUN TestRaft_RemoveLeader testLoggerAdapter verbose: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.509Z [INFO] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 Address:b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349} {Suffrage:Voter ID:server-000e1006-437f-af1b-8db7-148ccf97d74b Address:000e1006-437f-af1b-8db7-148ccf97d74b} {Suffrage:Voter ID:server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 Address:051c4c20-115e-e3ed-8769-6c28e69fb7f3}]" testing.go:149: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.509Z [INFO] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 Address:b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349} {Suffrage:Voter ID:server-000e1006-437f-af1b-8db7-148ccf97d74b Address:000e1006-437f-af1b-8db7-148ccf97d74b} {Suffrage:Voter ID:server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 Address:051c4c20-115e-e3ed-8769-6c28e69fb7f3}]" testLoggerAdapter verbose: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.510Z [INFO] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: entering follower state: follower="Node at b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 [Follower]" leader= testing.go:149: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.510Z [INFO] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: entering follower state: follower="Node at b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 [Follower]" leader= testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.510Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 Address:b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349} {Suffrage:Voter ID:server-000e1006-437f-af1b-8db7-148ccf97d74b Address:000e1006-437f-af1b-8db7-148ccf97d74b} {Suffrage:Voter ID:server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 Address:051c4c20-115e-e3ed-8769-6c28e69fb7f3}]" testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.510Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 Address:b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349} {Suffrage:Voter ID:server-000e1006-437f-af1b-8db7-148ccf97d74b Address:000e1006-437f-af1b-8db7-148ccf97d74b} {Suffrage:Voter ID:server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 Address:051c4c20-115e-e3ed-8769-6c28e69fb7f3}]" testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.510Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: entering follower state: follower="Node at 000e1006-437f-af1b-8db7-148ccf97d74b [Follower]" leader= testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.510Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: entering follower state: follower="Node at 000e1006-437f-af1b-8db7-148ccf97d74b [Follower]" leader= testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.510Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 Address:b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349} {Suffrage:Voter ID:server-000e1006-437f-af1b-8db7-148ccf97d74b Address:000e1006-437f-af1b-8db7-148ccf97d74b} {Suffrage:Voter ID:server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 Address:051c4c20-115e-e3ed-8769-6c28e69fb7f3}]" testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.510Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 Address:b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349} {Suffrage:Voter ID:server-000e1006-437f-af1b-8db7-148ccf97d74b Address:000e1006-437f-af1b-8db7-148ccf97d74b} {Suffrage:Voter ID:server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 Address:051c4c20-115e-e3ed-8769-6c28e69fb7f3}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.510Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:08.510Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.510Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: entering follower state: follower="Node at 051c4c20-115e-e3ed-8769-6c28e69fb7f3 [Follower]" leader= testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.510Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: entering follower state: follower="Node at 051c4c20-115e-e3ed-8769-6c28e69fb7f3 [Follower]" leader= testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [WARN] server-000e1006-437f-af1b-8db7-148ccf97d74b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [WARN] server-000e1006-437f-af1b-8db7-148ccf97d74b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: entering candidate state: node="Node at 000e1006-437f-af1b-8db7-148ccf97d74b [Candidate]" term=2 testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: entering candidate state: node="Node at 000e1006-437f-af1b-8db7-148ccf97d74b [Candidate]" term=2 testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: election won: tally=2 testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: election won: tally=2 testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: entering leader state: leader="Node at 000e1006-437f-af1b-8db7-148ccf97d74b [Leader]" testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: entering leader state: leader="Node at 000e1006-437f-af1b-8db7-148ccf97d74b [Leader]" testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: added peer, starting replication: peer=server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: added peer, starting replication: peer=server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: added peer, starting replication: peer=server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: added peer, starting replication: peer=server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.582Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: pipelining replication: peer="{Voter server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 051c4c20-115e-e3ed-8769-6c28e69fb7f3}" testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.582Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: pipelining replication: peer="{Voter server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 051c4c20-115e-e3ed-8769-6c28e69fb7f3}" testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.582Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: pipelining replication: peer="{Voter server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349}" testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.582Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: pipelining replication: peer="{Voter server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.687Z [INFO] stable state for Leader reached at 2021-08-25 09:28:08.582279522 +0000 UTC m=+8.346495411 (1 nodes), 71.405804ms from start of poll, 72.73966ms from cluster start. Timeout at 2021-08-25 09:28:08.687336438 +0000 UTC m=+8.451552407, 105.056996ms after stability testing.go:149: cluster: 2021-08-25T09:28:08.687Z [INFO] stable state for Leader reached at 2021-08-25 09:28:08.582279522 +0000 UTC m=+8.346495411 (1 nodes), 71.405804ms from start of poll, 72.73966ms from cluster start. Timeout at 2021-08-25 09:28:08.687336438 +0000 UTC m=+8.451552407, 105.056996ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.692Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:08.692Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.798Z [INFO] stable state for Follower reached at 2021-08-25 09:28:08.692850705 +0000 UTC m=+8.457066614 (2 nodes), 2.619µs from start of poll, 183.310863ms from cluster start. Timeout at 2021-08-25 09:28:08.797965836 +0000 UTC m=+8.562181825, 105.115211ms after stability testing.go:149: cluster: 2021-08-25T09:28:08.798Z [INFO] stable state for Follower reached at 2021-08-25 09:28:08.692850705 +0000 UTC m=+8.457066614 (2 nodes), 2.619µs from start of poll, 183.310863ms from cluster start. Timeout at 2021-08-25 09:28:08.797965836 +0000 UTC m=+8.562181825, 105.115211ms after stability testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.798Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: updating configuration: command=RemoveServer server-id=server-000e1006-437f-af1b-8db7-148ccf97d74b server-addr= servers="[{Suffrage:Voter ID:server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 Address:b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349} {Suffrage:Voter ID:server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 Address:051c4c20-115e-e3ed-8769-6c28e69fb7f3}]" testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.798Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: updating configuration: command=RemoveServer server-id=server-000e1006-437f-af1b-8db7-148ccf97d74b server-addr= servers="[{Suffrage:Voter ID:server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 Address:b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349} {Suffrage:Voter ID:server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 Address:051c4c20-115e-e3ed-8769-6c28e69fb7f3}]" testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.798Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: removed ourself, shutting down testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.798Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: removed ourself, shutting down testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.798Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: aborting pipeline replication: peer="{Voter server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 051c4c20-115e-e3ed-8769-6c28e69fb7f3}" testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.798Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: aborting pipeline replication: peer="{Voter server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 051c4c20-115e-e3ed-8769-6c28e69fb7f3}" testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.798Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: aborting pipeline replication: peer="{Voter server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349}" testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.798Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: aborting pipeline replication: peer="{Voter server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349}" testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.886Z [WARN] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: heartbeat timeout reached, starting election: last-leader=000e1006-437f-af1b-8db7-148ccf97d74b testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.886Z [WARN] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: heartbeat timeout reached, starting election: last-leader=000e1006-437f-af1b-8db7-148ccf97d74b testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.886Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: entering candidate state: node="Node at 051c4c20-115e-e3ed-8769-6c28e69fb7f3 [Candidate]" term=3 testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.886Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: entering candidate state: node="Node at 051c4c20-115e-e3ed-8769-6c28e69fb7f3 [Candidate]" term=3 testLoggerAdapter verbose: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.886Z [WARN] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: rejecting vote request since we have a leader: from=051c4c20-115e-e3ed-8769-6c28e69fb7f3 leader=000e1006-437f-af1b-8db7-148ccf97d74b testing.go:149: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.886Z [WARN] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: rejecting vote request since we have a leader: from=051c4c20-115e-e3ed-8769-6c28e69fb7f3 leader=000e1006-437f-af1b-8db7-148ccf97d74b testLoggerAdapter verbose: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.907Z [WARN] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: heartbeat timeout reached, starting election: last-leader=000e1006-437f-af1b-8db7-148ccf97d74b testing.go:149: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.907Z [WARN] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: heartbeat timeout reached, starting election: last-leader=000e1006-437f-af1b-8db7-148ccf97d74b testLoggerAdapter verbose: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.907Z [INFO] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: entering candidate state: node="Node at b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 [Candidate]" term=3 testing.go:149: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.907Z [INFO] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: entering candidate state: node="Node at b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 [Candidate]" term=3 testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.908Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: duplicate requestVote for same term: term=3 testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.908Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [WARN] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: Election timeout reached, restarting election testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [WARN] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: Election timeout reached, restarting election testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: entering candidate state: node="Node at 051c4c20-115e-e3ed-8769-6c28e69fb7f3 [Candidate]" term=4 testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: entering candidate state: node="Node at 051c4c20-115e-e3ed-8769-6c28e69fb7f3 [Candidate]" term=4 testLoggerAdapter verbose: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.953Z [INFO] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: entering follower state: follower="Node at b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 [Follower]" leader= testing.go:149: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.953Z [INFO] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: entering follower state: follower="Node at b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 [Follower]" leader= testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: election won: tally=2 testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: election won: tally=2 testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: entering leader state: leader="Node at 051c4c20-115e-e3ed-8769-6c28e69fb7f3 [Leader]" testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: entering leader state: leader="Node at 051c4c20-115e-e3ed-8769-6c28e69fb7f3 [Leader]" testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: added peer, starting replication: peer=server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: added peer, starting replication: peer=server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: pipelining replication: peer="{Voter server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349}" testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: pipelining replication: peer="{Voter server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.009Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:09.009Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.114Z [INFO] stable state for Leader reached at 2021-08-25 09:28:09.009637643 +0000 UTC m=+8.773853512 (1 nodes), 123.788µs from start of poll, 500.097761ms from cluster start. Timeout at 2021-08-25 09:28:09.114769392 +0000 UTC m=+8.878985421, 105.131909ms after stability testing.go:149: cluster: 2021-08-25T09:28:09.114Z [INFO] stable state for Leader reached at 2021-08-25 09:28:09.009637643 +0000 UTC m=+8.773853512 (1 nodes), 123.788µs from start of poll, 500.097761ms from cluster start. Timeout at 2021-08-25 09:28:09.114769392 +0000 UTC m=+8.878985421, 105.131909ms after stability testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:09.115Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: aborting pipeline replication: peer="{Voter server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349}" testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:09.115Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: aborting pipeline replication: peer="{Voter server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349}" --- PASS: TestRaft_RemoveLeader (0.61s) === RUN TestRaft_RemoveLeader_NoShutdown testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.118Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30230337-8642-d28a-9e21-942a5cb1c648 Address:30230337-8642-d28a-9e21-942a5cb1c648} {Suffrage:Voter ID:server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b Address:5f9267ed-84b2-25ec-98ff-9f2ed4e6394b} {Suffrage:Voter ID:server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 Address:8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31}]" testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.118Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30230337-8642-d28a-9e21-942a5cb1c648 Address:30230337-8642-d28a-9e21-942a5cb1c648} {Suffrage:Voter ID:server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b Address:5f9267ed-84b2-25ec-98ff-9f2ed4e6394b} {Suffrage:Voter ID:server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 Address:8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31}]" testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.118Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: entering follower state: follower="Node at 30230337-8642-d28a-9e21-942a5cb1c648 [Follower]" leader= testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.118Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: entering follower state: follower="Node at 30230337-8642-d28a-9e21-942a5cb1c648 [Follower]" leader= testLoggerAdapter verbose: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.118Z [INFO] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30230337-8642-d28a-9e21-942a5cb1c648 Address:30230337-8642-d28a-9e21-942a5cb1c648} {Suffrage:Voter ID:server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b Address:5f9267ed-84b2-25ec-98ff-9f2ed4e6394b} {Suffrage:Voter ID:server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 Address:8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31}]" testing.go:149: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.118Z [INFO] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30230337-8642-d28a-9e21-942a5cb1c648 Address:30230337-8642-d28a-9e21-942a5cb1c648} {Suffrage:Voter ID:server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b Address:5f9267ed-84b2-25ec-98ff-9f2ed4e6394b} {Suffrage:Voter ID:server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 Address:8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31}]" testLoggerAdapter verbose: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.118Z [INFO] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: entering follower state: follower="Node at 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b [Follower]" leader= testing.go:149: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.118Z [INFO] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: entering follower state: follower="Node at 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b [Follower]" leader= testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.119Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30230337-8642-d28a-9e21-942a5cb1c648 Address:30230337-8642-d28a-9e21-942a5cb1c648} {Suffrage:Voter ID:server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b Address:5f9267ed-84b2-25ec-98ff-9f2ed4e6394b} {Suffrage:Voter ID:server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 Address:8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31}]" testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.119Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30230337-8642-d28a-9e21-942a5cb1c648 Address:30230337-8642-d28a-9e21-942a5cb1c648} {Suffrage:Voter ID:server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b Address:5f9267ed-84b2-25ec-98ff-9f2ed4e6394b} {Suffrage:Voter ID:server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 Address:8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.119Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.119Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: entering follower state: follower="Node at 8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 [Follower]" leader= testing.go:149: cluster: 2021-08-25T09:28:09.119Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.119Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: entering follower state: follower="Node at 8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 [Follower]" leader= testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [WARN] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [WARN] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: entering candidate state: node="Node at 8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 [Candidate]" term=2 testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: entering candidate state: node="Node at 8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 [Candidate]" term=2 testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: election won: tally=2 testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: election won: tally=2 testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: entering leader state: leader="Node at 8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 [Leader]" testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: entering leader state: leader="Node at 8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 [Leader]" testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: added peer, starting replication: peer=server-30230337-8642-d28a-9e21-942a5cb1c648 testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: added peer, starting replication: peer=server-30230337-8642-d28a-9e21-942a5cb1c648 testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.187Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: added peer, starting replication: peer=server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.187Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: added peer, starting replication: peer=server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.187Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: pipelining replication: peer="{Voter server-30230337-8642-d28a-9e21-942a5cb1c648 30230337-8642-d28a-9e21-942a5cb1c648}" testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.187Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: pipelining replication: peer="{Voter server-30230337-8642-d28a-9e21-942a5cb1c648 30230337-8642-d28a-9e21-942a5cb1c648}" testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.187Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: pipelining replication: peer="{Voter server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}" testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.187Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: pipelining replication: peer="{Voter server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.292Z [INFO] stable state for Follower reached at 2021-08-25 09:28:09.187123548 +0000 UTC m=+8.951339476 (2 nodes), 67.818796ms from start of poll, 68.890557ms from cluster start. Timeout at 2021-08-25 09:28:09.292176504 +0000 UTC m=+9.056392493, 105.053017ms after stability testing.go:149: cluster: 2021-08-25T09:28:09.292Z [INFO] stable state for Follower reached at 2021-08-25 09:28:09.187123548 +0000 UTC m=+8.951339476 (2 nodes), 67.818796ms from start of poll, 68.890557ms from cluster start. Timeout at 2021-08-25 09:28:09.292176504 +0000 UTC m=+9.056392493, 105.053017ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.292Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:09.292Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.397Z [INFO] stable state for Leader reached at 2021-08-25 09:28:09.292426941 +0000 UTC m=+9.056642830 (1 nodes), 2.54µs from start of poll, 174.193911ms from cluster start. Timeout at 2021-08-25 09:28:09.397479657 +0000 UTC m=+9.161695566, 105.052736ms after stability testing.go:149: cluster: 2021-08-25T09:28:09.397Z [INFO] stable state for Leader reached at 2021-08-25 09:28:09.292426941 +0000 UTC m=+9.056642830 (1 nodes), 2.54µs from start of poll, 174.193911ms from cluster start. Timeout at 2021-08-25 09:28:09.397479657 +0000 UTC m=+9.161695566, 105.052736ms after stability testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.400Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: updating configuration: command=RemoveServer server-id=server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 server-addr= servers="[{Suffrage:Voter ID:server-30230337-8642-d28a-9e21-942a5cb1c648 Address:30230337-8642-d28a-9e21-942a5cb1c648} {Suffrage:Voter ID:server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b Address:5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}]" testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.400Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: updating configuration: command=RemoveServer server-id=server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 server-addr= servers="[{Suffrage:Voter ID:server-30230337-8642-d28a-9e21-942a5cb1c648 Address:30230337-8642-d28a-9e21-942a5cb1c648} {Suffrage:Voter ID:server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b Address:5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}]" testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.400Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: removed ourself, transitioning to follower testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.400Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: removed ourself, transitioning to follower testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.401Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: entering follower state: follower="Node at 8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 [Follower]" leader= testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.401Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: entering follower state: follower="Node at 8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 [Follower]" leader= testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.401Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: aborting pipeline replication: peer="{Voter server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}" testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.401Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: aborting pipeline replication: peer="{Voter server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}" testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.401Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: aborting pipeline replication: peer="{Voter server-30230337-8642-d28a-9e21-942a5cb1c648 30230337-8642-d28a-9e21-942a5cb1c648}" testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.401Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: aborting pipeline replication: peer="{Voter server-30230337-8642-d28a-9e21-942a5cb1c648 30230337-8642-d28a-9e21-942a5cb1c648}" testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.463Z [WARN] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: not part of stable configuration, aborting election testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.463Z [WARN] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: not part of stable configuration, aborting election testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.494Z [WARN] server-30230337-8642-d28a-9e21-942a5cb1c648: heartbeat timeout reached, starting election: last-leader=8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.494Z [WARN] server-30230337-8642-d28a-9e21-942a5cb1c648: heartbeat timeout reached, starting election: last-leader=8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.494Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: entering candidate state: node="Node at 30230337-8642-d28a-9e21-942a5cb1c648 [Candidate]" term=3 testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.494Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: entering candidate state: node="Node at 30230337-8642-d28a-9e21-942a5cb1c648 [Candidate]" term=3 testLoggerAdapter verbose: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.494Z [WARN] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: rejecting vote request since we have a leader: from=30230337-8642-d28a-9e21-942a5cb1c648 leader=8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 testing.go:149: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.494Z [WARN] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: rejecting vote request since we have a leader: from=30230337-8642-d28a-9e21-942a5cb1c648 leader=8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.506Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:09.506Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.518Z [WARN] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: heartbeat timeout reached, starting election: last-leader=8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 testing.go:149: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.518Z [WARN] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: heartbeat timeout reached, starting election: last-leader=8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 testLoggerAdapter verbose: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.518Z [INFO] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: entering candidate state: node="Node at 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b [Candidate]" term=3 testing.go:149: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.518Z [INFO] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: entering candidate state: node="Node at 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b [Candidate]" term=3 testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.519Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: duplicate requestVote for same term: term=3 testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.519Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [WARN] server-30230337-8642-d28a-9e21-942a5cb1c648: Election timeout reached, restarting election testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [WARN] server-30230337-8642-d28a-9e21-942a5cb1c648: Election timeout reached, restarting election testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: entering candidate state: node="Node at 30230337-8642-d28a-9e21-942a5cb1c648 [Candidate]" term=4 testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: entering candidate state: node="Node at 30230337-8642-d28a-9e21-942a5cb1c648 [Candidate]" term=4 testLoggerAdapter verbose: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.552Z [INFO] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: entering follower state: follower="Node at 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b [Follower]" leader= testing.go:149: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.552Z [INFO] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: entering follower state: follower="Node at 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b [Follower]" leader= testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: election won: tally=2 testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: election won: tally=2 testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: entering leader state: leader="Node at 30230337-8642-d28a-9e21-942a5cb1c648 [Leader]" testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: entering leader state: leader="Node at 30230337-8642-d28a-9e21-942a5cb1c648 [Leader]" testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: added peer, starting replication: peer=server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: added peer, starting replication: peer=server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.553Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: pipelining replication: peer="{Voter server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}" testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.553Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: pipelining replication: peer="{Voter server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.658Z [INFO] stable state for Leader reached at 2021-08-25 09:28:09.553221882 +0000 UTC m=+9.317437791 (1 nodes), 46.613027ms from start of poll, 434.988872ms from cluster start. Timeout at 2021-08-25 09:28:09.65836279 +0000 UTC m=+9.422578759, 105.140968ms after stability testing.go:149: cluster: 2021-08-25T09:28:09.658Z [INFO] stable state for Leader reached at 2021-08-25 09:28:09.553221882 +0000 UTC m=+9.317437791 (1 nodes), 46.613027ms from start of poll, 434.988872ms from cluster start. Timeout at 2021-08-25 09:28:09.65836279 +0000 UTC m=+9.422578759, 105.140968ms after stability testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.763Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: aborting pipeline replication: peer="{Voter server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}" testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.763Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: aborting pipeline replication: peer="{Voter server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}" --- PASS: TestRaft_RemoveLeader_NoShutdown (0.65s) === RUN TestRaft_RemoveFollower_SplitCluster testLoggerAdapter verbose: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:09.767Z [INFO] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-30d61c37-79cc-9c5d-feab-0046126d51ca Address:30d61c37-79cc-9c5d-feab-0046126d51ca} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" testing.go:149: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:09.767Z [INFO] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-30d61c37-79cc-9c5d-feab-0046126d51ca Address:30d61c37-79cc-9c5d-feab-0046126d51ca} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" testLoggerAdapter verbose: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:09.768Z [INFO] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: entering follower state: follower="Node at 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 [Follower]" leader= testing.go:149: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:09.768Z [INFO] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: entering follower state: follower="Node at 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 [Follower]" leader= testLoggerAdapter verbose: server-c7612fa6-4ceb-63b0-97ad-63dd16716550: 2021-08-25T09:28:09.768Z [INFO] server-c7612fa6-4ceb-63b0-97ad-63dd16716550: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-30d61c37-79cc-9c5d-feab-0046126d51ca Address:30d61c37-79cc-9c5d-feab-0046126d51ca} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" testing.go:149: server-c7612fa6-4ceb-63b0-97ad-63dd16716550: 2021-08-25T09:28:09.768Z [INFO] server-c7612fa6-4ceb-63b0-97ad-63dd16716550: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-30d61c37-79cc-9c5d-feab-0046126d51ca Address:30d61c37-79cc-9c5d-feab-0046126d51ca} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" testLoggerAdapter verbose: server-30d61c37-79cc-9c5d-feab-0046126d51ca: 2021-08-25T09:28:09.768Z [INFO] server-30d61c37-79cc-9c5d-feab-0046126d51ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-30d61c37-79cc-9c5d-feab-0046126d51ca Address:30d61c37-79cc-9c5d-feab-0046126d51ca} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" testing.go:149: server-30d61c37-79cc-9c5d-feab-0046126d51ca: 2021-08-25T09:28:09.768Z [INFO] server-30d61c37-79cc-9c5d-feab-0046126d51ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-30d61c37-79cc-9c5d-feab-0046126d51ca Address:30d61c37-79cc-9c5d-feab-0046126d51ca} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.768Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-30d61c37-79cc-9c5d-feab-0046126d51ca Address:30d61c37-79cc-9c5d-feab-0046126d51ca} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.768Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-30d61c37-79cc-9c5d-feab-0046126d51ca Address:30d61c37-79cc-9c5d-feab-0046126d51ca} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.768Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:09.768Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c7612fa6-4ceb-63b0-97ad-63dd16716550: 2021-08-25T09:28:09.769Z [INFO] server-c7612fa6-4ceb-63b0-97ad-63dd16716550: entering follower state: follower="Node at c7612fa6-4ceb-63b0-97ad-63dd16716550 [Follower]" leader= testing.go:149: server-c7612fa6-4ceb-63b0-97ad-63dd16716550: 2021-08-25T09:28:09.769Z [INFO] server-c7612fa6-4ceb-63b0-97ad-63dd16716550: entering follower state: follower="Node at c7612fa6-4ceb-63b0-97ad-63dd16716550 [Follower]" leader= testLoggerAdapter verbose: server-30d61c37-79cc-9c5d-feab-0046126d51ca: 2021-08-25T09:28:09.769Z [INFO] server-30d61c37-79cc-9c5d-feab-0046126d51ca: entering follower state: follower="Node at 30d61c37-79cc-9c5d-feab-0046126d51ca [Follower]" leader= testing.go:149: server-30d61c37-79cc-9c5d-feab-0046126d51ca: 2021-08-25T09:28:09.769Z [INFO] server-30d61c37-79cc-9c5d-feab-0046126d51ca: entering follower state: follower="Node at 30d61c37-79cc-9c5d-feab-0046126d51ca [Follower]" leader= testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.769Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: entering follower state: follower="Node at fd8b4f14-3588-16aa-f3ea-cff79821e401 [Follower]" leader= testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.769Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: entering follower state: follower="Node at fd8b4f14-3588-16aa-f3ea-cff79821e401 [Follower]" leader= testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.831Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.831Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.831Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: entering candidate state: node="Node at fd8b4f14-3588-16aa-f3ea-cff79821e401 [Candidate]" term=2 testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.831Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: entering candidate state: node="Node at fd8b4f14-3588-16aa-f3ea-cff79821e401 [Candidate]" term=2 testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.831Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: election won: tally=3 testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.831Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: election won: tally=3 testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: entering leader state: leader="Node at fd8b4f14-3588-16aa-f3ea-cff79821e401 [Leader]" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: entering leader state: leader="Node at fd8b4f14-3588-16aa-f3ea-cff79821e401 [Leader]" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: added peer, starting replication: peer=server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: added peer, starting replication: peer=server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: added peer, starting replication: peer=server-c7612fa6-4ceb-63b0-97ad-63dd16716550 testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: added peer, starting replication: peer=server-c7612fa6-4ceb-63b0-97ad-63dd16716550 testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: added peer, starting replication: peer=server-30d61c37-79cc-9c5d-feab-0046126d51ca testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: added peer, starting replication: peer=server-30d61c37-79cc-9c5d-feab-0046126d51ca testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: pipelining replication: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: pipelining replication: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: pipelining replication: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: pipelining replication: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: pipelining replication: peer="{Voter server-30d61c37-79cc-9c5d-feab-0046126d51ca 30d61c37-79cc-9c5d-feab-0046126d51ca}" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: pipelining replication: peer="{Voter server-30d61c37-79cc-9c5d-feab-0046126d51ca 30d61c37-79cc-9c5d-feab-0046126d51ca}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.937Z [INFO] stable state for Leader reached at 2021-08-25 09:28:09.832253849 +0000 UTC m=+9.596469738 (1 nodes), 63.244242ms from start of poll, 64.715725ms from cluster start. Timeout at 2021-08-25 09:28:09.937392377 +0000 UTC m=+9.701608406, 105.138668ms after stability testing.go:149: cluster: 2021-08-25T09:28:09.937Z [INFO] stable state for Leader reached at 2021-08-25 09:28:09.832253849 +0000 UTC m=+9.596469738 (1 nodes), 63.244242ms from start of poll, 64.715725ms from cluster start. Timeout at 2021-08-25 09:28:09.937392377 +0000 UTC m=+9.701608406, 105.138668ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.042Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:10.042Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.148Z [INFO] stable state for Follower reached at 2021-08-25 09:28:10.042953807 +0000 UTC m=+9.807169696 (3 nodes), 3.24µs from start of poll, 275.415683ms from cluster start. Timeout at 2021-08-25 09:28:10.148092315 +0000 UTC m=+9.912308264, 105.138568ms after stability testing.go:149: cluster: 2021-08-25T09:28:10.148Z [INFO] stable state for Follower reached at 2021-08-25 09:28:10.042953807 +0000 UTC m=+9.807169696 (3 nodes), 3.24µs from start of poll, 275.415683ms from cluster start. Timeout at 2021-08-25 09:28:10.148092315 +0000 UTC m=+9.912308264, 105.138568ms after stability testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: updating configuration: command=RemoveServer server-id=server-30d61c37-79cc-9c5d-feab-0046126d51ca server-addr= servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: updating configuration: command=RemoveServer server-id=server-30d61c37-79cc-9c5d-feab-0046126d51ca server-addr= servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: removed peer, stopping replication: peer=server-30d61c37-79cc-9c5d-feab-0046126d51ca last-index=3 testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: removed peer, stopping replication: peer=server-30d61c37-79cc-9c5d-feab-0046126d51ca last-index=3 testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to pipeline appendEntries: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="append pipeline closed" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to pipeline appendEntries: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="append pipeline closed" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: aborting pipeline replication: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: aborting pipeline replication: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to pipeline appendEntries: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="append pipeline closed" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to pipeline appendEntries: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="append pipeline closed" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: aborting pipeline replication: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: aborting pipeline replication: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: aborting pipeline replication: peer="{Voter server-30d61c37-79cc-9c5d-feab-0046126d51ca 30d61c37-79cc-9c5d-feab-0046126d51ca}" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: aborting pipeline replication: peer="{Voter server-30d61c37-79cc-9c5d-feab-0046126d51ca 30d61c37-79cc-9c5d-feab-0046126d51ca}" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.149Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=c7612fa6-4ceb-63b0-97ad-63dd16716550 error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.149Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=c7612fa6-4ceb-63b0-97ad-63dd16716550 error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.149Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.149Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.154Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.154Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.157Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.157Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.165Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=c7612fa6-4ceb-63b0-97ad-63dd16716550 error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.165Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=c7612fa6-4ceb-63b0-97ad-63dd16716550 error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.170Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.170Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.173Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.173Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.176Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.176Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.183Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=c7612fa6-4ceb-63b0-97ad-63dd16716550 error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.183Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=c7612fa6-4ceb-63b0-97ad-63dd16716550 error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.186Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.186Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.191Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to contact: server-id=server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 time=50.103124ms testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.191Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to contact: server-id=server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 time=50.103124ms testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.193Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.193Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.193Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.193Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" testLoggerAdapter verbose: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:10.195Z [WARN] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: heartbeat timeout reached, starting election: last-leader=fd8b4f14-3588-16aa-f3ea-cff79821e401 testing.go:149: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:10.195Z [WARN] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: heartbeat timeout reached, starting election: last-leader=fd8b4f14-3588-16aa-f3ea-cff79821e401 testLoggerAdapter verbose: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:10.195Z [INFO] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: entering candidate state: node="Node at 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 [Candidate]" term=3 testing.go:149: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:10.195Z [INFO] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: entering candidate state: node="Node at 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 [Candidate]" term=3 testLoggerAdapter verbose: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:10.195Z [ERROR] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: failed to make requestVote RPC: target="{Voter server-fd8b4f14-3588-16aa-f3ea-cff79821e401 fd8b4f14-3588-16aa-f3ea-cff79821e401}" error="failed to connect to peer: fd8b4f14-3588-16aa-f3ea-cff79821e401" testing.go:149: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:10.195Z [ERROR] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: failed to make requestVote RPC: target="{Voter server-fd8b4f14-3588-16aa-f3ea-cff79821e401 fd8b4f14-3588-16aa-f3ea-cff79821e401}" error="failed to connect to peer: fd8b4f14-3588-16aa-f3ea-cff79821e401" testLoggerAdapter verbose: server-c7612fa6-4ceb-63b0-97ad-63dd16716550: 2021-08-25T09:28:10.195Z [WARN] server-c7612fa6-4ceb-63b0-97ad-63dd16716550: rejecting vote request since we have a leader: from=1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 leader=fd8b4f14-3588-16aa-f3ea-cff79821e401 testing.go:149: server-c7612fa6-4ceb-63b0-97ad-63dd16716550: 2021-08-25T09:28:10.195Z [WARN] server-c7612fa6-4ceb-63b0-97ad-63dd16716550: rejecting vote request since we have a leader: from=1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 leader=fd8b4f14-3588-16aa-f3ea-cff79821e401 testLoggerAdapter verbose: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:10.195Z [ERROR] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: failed to make requestVote RPC: target="{Voter server-30d61c37-79cc-9c5d-feab-0046126d51ca 30d61c37-79cc-9c5d-feab-0046126d51ca}" error="failed to connect to peer: 30d61c37-79cc-9c5d-feab-0046126d51ca" testing.go:149: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:10.195Z [ERROR] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: failed to make requestVote RPC: target="{Voter server-30d61c37-79cc-9c5d-feab-0046126d51ca 30d61c37-79cc-9c5d-feab-0046126d51ca}" error="failed to connect to peer: 30d61c37-79cc-9c5d-feab-0046126d51ca" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to contact: server-id=server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 time=60.376569ms testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to contact: server-id=server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 time=60.376569ms testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to contact: server-id=server-c7612fa6-4ceb-63b0-97ad-63dd16716550 time=55.241106ms testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to contact: server-id=server-c7612fa6-4ceb-63b0-97ad-63dd16716550 time=55.241106ms testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to contact quorum of nodes, stepping down testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: entering follower state: follower="Node at fd8b4f14-3588-16aa-f3ea-cff79821e401 [Follower]" leader= testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: entering follower state: follower="Node at fd8b4f14-3588-16aa-f3ea-cff79821e401 [Follower]" leader= testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" --- PASS: TestRaft_RemoveFollower_SplitCluster (0.44s) === RUN TestRaft_AddKnownPeer testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.205Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 Address:4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15} {Suffrage:Voter ID:server-09a05e3a-97e3-9542-8e3d-14ac482b9432 Address:09a05e3a-97e3-9542-8e3d-14ac482b9432} {Suffrage:Voter ID:server-be21088c-02b5-ae20-1bc8-d13bc655a613 Address:be21088c-02b5-ae20-1bc8-d13bc655a613}]" testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.205Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 Address:4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15} {Suffrage:Voter ID:server-09a05e3a-97e3-9542-8e3d-14ac482b9432 Address:09a05e3a-97e3-9542-8e3d-14ac482b9432} {Suffrage:Voter ID:server-be21088c-02b5-ae20-1bc8-d13bc655a613 Address:be21088c-02b5-ae20-1bc8-d13bc655a613}]" testLoggerAdapter verbose: server-09a05e3a-97e3-9542-8e3d-14ac482b9432: 2021-08-25T09:28:10.206Z [INFO] server-09a05e3a-97e3-9542-8e3d-14ac482b9432: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 Address:4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15} {Suffrage:Voter ID:server-09a05e3a-97e3-9542-8e3d-14ac482b9432 Address:09a05e3a-97e3-9542-8e3d-14ac482b9432} {Suffrage:Voter ID:server-be21088c-02b5-ae20-1bc8-d13bc655a613 Address:be21088c-02b5-ae20-1bc8-d13bc655a613}]" testing.go:149: server-09a05e3a-97e3-9542-8e3d-14ac482b9432: 2021-08-25T09:28:10.206Z [INFO] server-09a05e3a-97e3-9542-8e3d-14ac482b9432: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 Address:4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15} {Suffrage:Voter ID:server-09a05e3a-97e3-9542-8e3d-14ac482b9432 Address:09a05e3a-97e3-9542-8e3d-14ac482b9432} {Suffrage:Voter ID:server-be21088c-02b5-ae20-1bc8-d13bc655a613 Address:be21088c-02b5-ae20-1bc8-d13bc655a613}]" testLoggerAdapter verbose: server-be21088c-02b5-ae20-1bc8-d13bc655a613: 2021-08-25T09:28:10.206Z [INFO] server-be21088c-02b5-ae20-1bc8-d13bc655a613: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 Address:4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15} {Suffrage:Voter ID:server-09a05e3a-97e3-9542-8e3d-14ac482b9432 Address:09a05e3a-97e3-9542-8e3d-14ac482b9432} {Suffrage:Voter ID:server-be21088c-02b5-ae20-1bc8-d13bc655a613 Address:be21088c-02b5-ae20-1bc8-d13bc655a613}]" testing.go:149: server-be21088c-02b5-ae20-1bc8-d13bc655a613: 2021-08-25T09:28:10.206Z [INFO] server-be21088c-02b5-ae20-1bc8-d13bc655a613: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 Address:4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15} {Suffrage:Voter ID:server-09a05e3a-97e3-9542-8e3d-14ac482b9432 Address:09a05e3a-97e3-9542-8e3d-14ac482b9432} {Suffrage:Voter ID:server-be21088c-02b5-ae20-1bc8-d13bc655a613 Address:be21088c-02b5-ae20-1bc8-d13bc655a613}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.206Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:10.206Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.206Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: entering follower state: follower="Node at 4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 [Follower]" leader= testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.206Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: entering follower state: follower="Node at 4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 [Follower]" leader= testLoggerAdapter verbose: server-be21088c-02b5-ae20-1bc8-d13bc655a613: 2021-08-25T09:28:10.206Z [INFO] server-be21088c-02b5-ae20-1bc8-d13bc655a613: entering follower state: follower="Node at be21088c-02b5-ae20-1bc8-d13bc655a613 [Follower]" leader= testLoggerAdapter verbose: server-09a05e3a-97e3-9542-8e3d-14ac482b9432: 2021-08-25T09:28:10.206Z [INFO] server-09a05e3a-97e3-9542-8e3d-14ac482b9432: entering follower state: follower="Node at 09a05e3a-97e3-9542-8e3d-14ac482b9432 [Follower]" leader= testing.go:149: server-be21088c-02b5-ae20-1bc8-d13bc655a613: 2021-08-25T09:28:10.206Z [INFO] server-be21088c-02b5-ae20-1bc8-d13bc655a613: entering follower state: follower="Node at be21088c-02b5-ae20-1bc8-d13bc655a613 [Follower]" leader= testing.go:149: server-09a05e3a-97e3-9542-8e3d-14ac482b9432: 2021-08-25T09:28:10.206Z [INFO] server-09a05e3a-97e3-9542-8e3d-14ac482b9432: entering follower state: follower="Node at 09a05e3a-97e3-9542-8e3d-14ac482b9432 [Follower]" leader= testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [WARN] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [WARN] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: entering candidate state: node="Node at 4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 [Candidate]" term=2 testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: entering candidate state: node="Node at 4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 [Candidate]" term=2 testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: election won: tally=2 testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: election won: tally=2 testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: entering leader state: leader="Node at 4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 [Leader]" testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: entering leader state: leader="Node at 4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 [Leader]" testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: added peer, starting replication: peer=server-09a05e3a-97e3-9542-8e3d-14ac482b9432 testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: added peer, starting replication: peer=server-09a05e3a-97e3-9542-8e3d-14ac482b9432 testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: added peer, starting replication: peer=server-be21088c-02b5-ae20-1bc8-d13bc655a613 testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: added peer, starting replication: peer=server-be21088c-02b5-ae20-1bc8-d13bc655a613 testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.261Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: pipelining replication: peer="{Voter server-09a05e3a-97e3-9542-8e3d-14ac482b9432 09a05e3a-97e3-9542-8e3d-14ac482b9432}" testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.261Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: pipelining replication: peer="{Voter server-09a05e3a-97e3-9542-8e3d-14ac482b9432 09a05e3a-97e3-9542-8e3d-14ac482b9432}" testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.261Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: pipelining replication: peer="{Voter server-be21088c-02b5-ae20-1bc8-d13bc655a613 be21088c-02b5-ae20-1bc8-d13bc655a613}" testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.261Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: pipelining replication: peer="{Voter server-be21088c-02b5-ae20-1bc8-d13bc655a613 be21088c-02b5-ae20-1bc8-d13bc655a613}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.366Z [INFO] stable state for Leader reached at 2021-08-25 09:28:10.260976684 +0000 UTC m=+10.025192593 (1 nodes), 54.490176ms from start of poll, 55.579355ms from cluster start. Timeout at 2021-08-25 09:28:10.366007922 +0000 UTC m=+10.130223911, 105.031318ms after stability testing.go:149: cluster: 2021-08-25T09:28:10.366Z [INFO] stable state for Leader reached at 2021-08-25 09:28:10.260976684 +0000 UTC m=+10.025192593 (1 nodes), 54.490176ms from start of poll, 55.579355ms from cluster start. Timeout at 2021-08-25 09:28:10.366007922 +0000 UTC m=+10.130223911, 105.031318ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.366Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:10.366Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.471Z [INFO] stable state for Follower reached at 2021-08-25 09:28:10.366266798 +0000 UTC m=+10.130482687 (2 nodes), 2.9µs from start of poll, 160.869449ms from cluster start. Timeout at 2021-08-25 09:28:10.471380229 +0000 UTC m=+10.235596198, 105.113511ms after stability testing.go:149: cluster: 2021-08-25T09:28:10.471Z [INFO] stable state for Follower reached at 2021-08-25 09:28:10.366266798 +0000 UTC m=+10.130482687 (2 nodes), 2.9µs from start of poll, 160.869449ms from cluster start. Timeout at 2021-08-25 09:28:10.471380229 +0000 UTC m=+10.235596198, 105.113511ms after stability testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.471Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: updating configuration: command=AddStaging server-id=server-09a05e3a-97e3-9542-8e3d-14ac482b9432 server-addr=09a05e3a-97e3-9542-8e3d-14ac482b9432 servers="[{Suffrage:Voter ID:server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 Address:4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15} {Suffrage:Voter ID:server-09a05e3a-97e3-9542-8e3d-14ac482b9432 Address:09a05e3a-97e3-9542-8e3d-14ac482b9432} {Suffrage:Voter ID:server-be21088c-02b5-ae20-1bc8-d13bc655a613 Address:be21088c-02b5-ae20-1bc8-d13bc655a613}]" testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.471Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: updating configuration: command=AddStaging server-id=server-09a05e3a-97e3-9542-8e3d-14ac482b9432 server-addr=09a05e3a-97e3-9542-8e3d-14ac482b9432 servers="[{Suffrage:Voter ID:server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 Address:4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15} {Suffrage:Voter ID:server-09a05e3a-97e3-9542-8e3d-14ac482b9432 Address:09a05e3a-97e3-9542-8e3d-14ac482b9432} {Suffrage:Voter ID:server-be21088c-02b5-ae20-1bc8-d13bc655a613 Address:be21088c-02b5-ae20-1bc8-d13bc655a613}]" testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.472Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: aborting pipeline replication: peer="{Voter server-be21088c-02b5-ae20-1bc8-d13bc655a613 be21088c-02b5-ae20-1bc8-d13bc655a613}" testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.472Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: aborting pipeline replication: peer="{Voter server-be21088c-02b5-ae20-1bc8-d13bc655a613 be21088c-02b5-ae20-1bc8-d13bc655a613}" testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.472Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: aborting pipeline replication: peer="{Voter server-09a05e3a-97e3-9542-8e3d-14ac482b9432 09a05e3a-97e3-9542-8e3d-14ac482b9432}" testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.472Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: aborting pipeline replication: peer="{Voter server-09a05e3a-97e3-9542-8e3d-14ac482b9432 09a05e3a-97e3-9542-8e3d-14ac482b9432}" --- PASS: TestRaft_AddKnownPeer (0.27s) === RUN TestRaft_RemoveUnknownPeer testLoggerAdapter verbose: server-e25a3674-09ad-4651-ba9f-4354641b41fe: 2021-08-25T09:28:10.477Z [INFO] server-e25a3674-09ad-4651-ba9f-4354641b41fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e25a3674-09ad-4651-ba9f-4354641b41fe Address:e25a3674-09ad-4651-ba9f-4354641b41fe} {Suffrage:Voter ID:server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 Address:2b3b2530-7483-b1f2-759a-5c1ca58e4eb4} {Suffrage:Voter ID:server-0980da88-1e39-d44e-6aa0-522298e67f99 Address:0980da88-1e39-d44e-6aa0-522298e67f99}]" testing.go:149: server-e25a3674-09ad-4651-ba9f-4354641b41fe: 2021-08-25T09:28:10.477Z [INFO] server-e25a3674-09ad-4651-ba9f-4354641b41fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e25a3674-09ad-4651-ba9f-4354641b41fe Address:e25a3674-09ad-4651-ba9f-4354641b41fe} {Suffrage:Voter ID:server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 Address:2b3b2530-7483-b1f2-759a-5c1ca58e4eb4} {Suffrage:Voter ID:server-0980da88-1e39-d44e-6aa0-522298e67f99 Address:0980da88-1e39-d44e-6aa0-522298e67f99}]" testLoggerAdapter verbose: server-e25a3674-09ad-4651-ba9f-4354641b41fe: 2021-08-25T09:28:10.477Z [INFO] server-e25a3674-09ad-4651-ba9f-4354641b41fe: entering follower state: follower="Node at e25a3674-09ad-4651-ba9f-4354641b41fe [Follower]" leader= testing.go:149: server-e25a3674-09ad-4651-ba9f-4354641b41fe: 2021-08-25T09:28:10.477Z [INFO] server-e25a3674-09ad-4651-ba9f-4354641b41fe: entering follower state: follower="Node at e25a3674-09ad-4651-ba9f-4354641b41fe [Follower]" leader= testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.477Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e25a3674-09ad-4651-ba9f-4354641b41fe Address:e25a3674-09ad-4651-ba9f-4354641b41fe} {Suffrage:Voter ID:server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 Address:2b3b2530-7483-b1f2-759a-5c1ca58e4eb4} {Suffrage:Voter ID:server-0980da88-1e39-d44e-6aa0-522298e67f99 Address:0980da88-1e39-d44e-6aa0-522298e67f99}]" testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.477Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e25a3674-09ad-4651-ba9f-4354641b41fe Address:e25a3674-09ad-4651-ba9f-4354641b41fe} {Suffrage:Voter ID:server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 Address:2b3b2530-7483-b1f2-759a-5c1ca58e4eb4} {Suffrage:Voter ID:server-0980da88-1e39-d44e-6aa0-522298e67f99 Address:0980da88-1e39-d44e-6aa0-522298e67f99}]" testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.477Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: entering follower state: follower="Node at 2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 [Follower]" leader= testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.477Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: entering follower state: follower="Node at 2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 [Follower]" leader= testLoggerAdapter verbose: server-0980da88-1e39-d44e-6aa0-522298e67f99: 2021-08-25T09:28:10.477Z [INFO] server-0980da88-1e39-d44e-6aa0-522298e67f99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e25a3674-09ad-4651-ba9f-4354641b41fe Address:e25a3674-09ad-4651-ba9f-4354641b41fe} {Suffrage:Voter ID:server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 Address:2b3b2530-7483-b1f2-759a-5c1ca58e4eb4} {Suffrage:Voter ID:server-0980da88-1e39-d44e-6aa0-522298e67f99 Address:0980da88-1e39-d44e-6aa0-522298e67f99}]" testing.go:149: server-0980da88-1e39-d44e-6aa0-522298e67f99: 2021-08-25T09:28:10.477Z [INFO] server-0980da88-1e39-d44e-6aa0-522298e67f99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e25a3674-09ad-4651-ba9f-4354641b41fe Address:e25a3674-09ad-4651-ba9f-4354641b41fe} {Suffrage:Voter ID:server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 Address:2b3b2530-7483-b1f2-759a-5c1ca58e4eb4} {Suffrage:Voter ID:server-0980da88-1e39-d44e-6aa0-522298e67f99 Address:0980da88-1e39-d44e-6aa0-522298e67f99}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.478Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-0980da88-1e39-d44e-6aa0-522298e67f99: 2021-08-25T09:28:10.478Z [INFO] server-0980da88-1e39-d44e-6aa0-522298e67f99: entering follower state: follower="Node at 0980da88-1e39-d44e-6aa0-522298e67f99 [Follower]" leader= testing.go:149: cluster: 2021-08-25T09:28:10.478Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-0980da88-1e39-d44e-6aa0-522298e67f99: 2021-08-25T09:28:10.478Z [INFO] server-0980da88-1e39-d44e-6aa0-522298e67f99: entering follower state: follower="Node at 0980da88-1e39-d44e-6aa0-522298e67f99 [Follower]" leader= testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.551Z [WARN] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.551Z [WARN] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.551Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: entering candidate state: node="Node at 2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 [Candidate]" term=2 testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.551Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: entering candidate state: node="Node at 2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 [Candidate]" term=2 testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: election won: tally=2 testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: election won: tally=2 testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: entering leader state: leader="Node at 2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 [Leader]" testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: entering leader state: leader="Node at 2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 [Leader]" testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: added peer, starting replication: peer=server-e25a3674-09ad-4651-ba9f-4354641b41fe testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: added peer, starting replication: peer=server-e25a3674-09ad-4651-ba9f-4354641b41fe testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: added peer, starting replication: peer=server-0980da88-1e39-d44e-6aa0-522298e67f99 testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: added peer, starting replication: peer=server-0980da88-1e39-d44e-6aa0-522298e67f99 testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: pipelining replication: peer="{Voter server-0980da88-1e39-d44e-6aa0-522298e67f99 0980da88-1e39-d44e-6aa0-522298e67f99}" testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: pipelining replication: peer="{Voter server-0980da88-1e39-d44e-6aa0-522298e67f99 0980da88-1e39-d44e-6aa0-522298e67f99}" testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: pipelining replication: peer="{Voter server-e25a3674-09ad-4651-ba9f-4354641b41fe e25a3674-09ad-4651-ba9f-4354641b41fe}" testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: pipelining replication: peer="{Voter server-e25a3674-09ad-4651-ba9f-4354641b41fe e25a3674-09ad-4651-ba9f-4354641b41fe}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.657Z [INFO] stable state for Leader reached at 2021-08-25 09:28:10.552299848 +0000 UTC m=+10.316515737 (1 nodes), 74.098513ms from start of poll, 75.430629ms from cluster start. Timeout at 2021-08-25 09:28:10.657328767 +0000 UTC m=+10.421544736, 105.028999ms after stability testing.go:149: cluster: 2021-08-25T09:28:10.657Z [INFO] stable state for Leader reached at 2021-08-25 09:28:10.552299848 +0000 UTC m=+10.316515737 (1 nodes), 74.098513ms from start of poll, 75.430629ms from cluster start. Timeout at 2021-08-25 09:28:10.657328767 +0000 UTC m=+10.421544736, 105.028999ms after stability testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.657Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: updating configuration: command=RemoveServer server-id=9cb5e7c3-f4fb-cfc9-991c-7296879b73b6 server-addr= servers="[{Suffrage:Voter ID:server-e25a3674-09ad-4651-ba9f-4354641b41fe Address:e25a3674-09ad-4651-ba9f-4354641b41fe} {Suffrage:Voter ID:server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 Address:2b3b2530-7483-b1f2-759a-5c1ca58e4eb4} {Suffrage:Voter ID:server-0980da88-1e39-d44e-6aa0-522298e67f99 Address:0980da88-1e39-d44e-6aa0-522298e67f99}]" testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.657Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: updating configuration: command=RemoveServer server-id=9cb5e7c3-f4fb-cfc9-991c-7296879b73b6 server-addr= servers="[{Suffrage:Voter ID:server-e25a3674-09ad-4651-ba9f-4354641b41fe Address:e25a3674-09ad-4651-ba9f-4354641b41fe} {Suffrage:Voter ID:server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 Address:2b3b2530-7483-b1f2-759a-5c1ca58e4eb4} {Suffrage:Voter ID:server-0980da88-1e39-d44e-6aa0-522298e67f99 Address:0980da88-1e39-d44e-6aa0-522298e67f99}]" testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.658Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: aborting pipeline replication: peer="{Voter server-0980da88-1e39-d44e-6aa0-522298e67f99 0980da88-1e39-d44e-6aa0-522298e67f99}" testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.658Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: aborting pipeline replication: peer="{Voter server-0980da88-1e39-d44e-6aa0-522298e67f99 0980da88-1e39-d44e-6aa0-522298e67f99}" testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.658Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: aborting pipeline replication: peer="{Voter server-e25a3674-09ad-4651-ba9f-4354641b41fe e25a3674-09ad-4651-ba9f-4354641b41fe}" testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.658Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: aborting pipeline replication: peer="{Voter server-e25a3674-09ad-4651-ba9f-4354641b41fe e25a3674-09ad-4651-ba9f-4354641b41fe}" --- PASS: TestRaft_RemoveUnknownPeer (0.18s) === RUN TestRaft_SnapshotRestore testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.660Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 Address:c543544b-52e7-7e6d-5e5d-2c31bea1c4e3}]" testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.660Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 Address:c543544b-52e7-7e6d-5e5d-2c31bea1c4e3}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.660Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:10.660Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.660Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: entering follower state: follower="Node at c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 [Follower]" leader= testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.660Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: entering follower state: follower="Node at c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 [Follower]" leader= testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.718Z [WARN] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.718Z [WARN] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.718Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: entering candidate state: node="Node at c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 [Candidate]" term=2 testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.718Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: entering candidate state: node="Node at c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 [Candidate]" term=2 testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.718Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: election won: tally=1 testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.718Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: election won: tally=1 testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.718Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: entering leader state: leader="Node at c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 [Leader]" testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.718Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: entering leader state: leader="Node at c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 [Leader]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.823Z [INFO] stable state for Leader reached at 2021-08-25 09:28:10.718598352 +0000 UTC m=+10.482814261 (1 nodes), 57.789209ms from start of poll, 58.163874ms from cluster start. Timeout at 2021-08-25 09:28:10.823727302 +0000 UTC m=+10.587943331, 105.12907ms after stability testing.go:149: cluster: 2021-08-25T09:28:10.823Z [INFO] stable state for Leader reached at 2021-08-25 09:28:10.718598352 +0000 UTC m=+10.482814261 (1 nodes), 57.789209ms from start of poll, 58.163874ms from cluster start. Timeout at 2021-08-25 09:28:10.823727302 +0000 UTC m=+10.587943331, 105.12907ms after stability testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.828Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: starting snapshot up to: index=102 testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.828Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: starting snapshot up to: index=102 testing.go:153: 2021-08-25T09:28:10.828Z [INFO] creating new snapshot: path=/tmp/raft457250717/snapshots/2-102-1629883690828.tmp testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.846Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: compacting logs: from=1 to=92 testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.846Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.847Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: snapshot complete up to: index=102 testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.847Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: snapshot complete up to: index=102 testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.847Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: restored from snapshot: id=2-102-1629883690828 testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.847Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: restored from snapshot: id=2-102-1629883690828 testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.848Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 Address:c543544b-52e7-7e6d-5e5d-2c31bea1c4e3}]" testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.848Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 Address:c543544b-52e7-7e6d-5e5d-2c31bea1c4e3}]" --- PASS: TestRaft_SnapshotRestore (0.19s) === RUN TestRaft_NoRestoreOnStart testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.849Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6af65f82-5092-4322-d891-3af4a67edd70 Address:6af65f82-5092-4322-d891-3af4a67edd70}]" testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.849Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6af65f82-5092-4322-d891-3af4a67edd70 Address:6af65f82-5092-4322-d891-3af4a67edd70}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.849Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:10.849Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.850Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: entering follower state: follower="Node at 6af65f82-5092-4322-d891-3af4a67edd70 [Follower]" leader= testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.850Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: entering follower state: follower="Node at 6af65f82-5092-4322-d891-3af4a67edd70 [Follower]" leader= testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.941Z [WARN] server-6af65f82-5092-4322-d891-3af4a67edd70: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.941Z [WARN] server-6af65f82-5092-4322-d891-3af4a67edd70: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.941Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: entering candidate state: node="Node at 6af65f82-5092-4322-d891-3af4a67edd70 [Candidate]" term=2 testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.941Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: entering candidate state: node="Node at 6af65f82-5092-4322-d891-3af4a67edd70 [Candidate]" term=2 testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.941Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: election won: tally=1 testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.941Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: election won: tally=1 testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.941Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: entering leader state: leader="Node at 6af65f82-5092-4322-d891-3af4a67edd70 [Leader]" testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.941Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: entering leader state: leader="Node at 6af65f82-5092-4322-d891-3af4a67edd70 [Leader]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:11.047Z [INFO] stable state for Leader reached at 2021-08-25 09:28:10.942114839 +0000 UTC m=+10.706330748 (1 nodes), 92.057145ms from start of poll, 92.485225ms from cluster start. Timeout at 2021-08-25 09:28:11.047146717 +0000 UTC m=+10.811362666, 105.031918ms after stability testing.go:149: cluster: 2021-08-25T09:28:11.047Z [INFO] stable state for Leader reached at 2021-08-25 09:28:10.942114839 +0000 UTC m=+10.706330748 (1 nodes), 92.057145ms from start of poll, 92.485225ms from cluster start. Timeout at 2021-08-25 09:28:11.047146717 +0000 UTC m=+10.811362666, 105.031918ms after stability testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.049Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: starting snapshot up to: index=102 testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.049Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: starting snapshot up to: index=102 testing.go:153: 2021-08-25T09:28:11.049Z [INFO] creating new snapshot: path=/tmp/raft819490775/snapshots/2-102-1629883691049.tmp testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.076Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: compacting logs: from=1 to=92 testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.076Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.076Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: snapshot complete up to: index=102 testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.076Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: snapshot complete up to: index=102 testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.076Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6af65f82-5092-4322-d891-3af4a67edd70 Address:6af65f82-5092-4322-d891-3af4a67edd70}]" testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.076Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6af65f82-5092-4322-d891-3af4a67edd70 Address:6af65f82-5092-4322-d891-3af4a67edd70}]" --- PASS: TestRaft_NoRestoreOnStart (0.23s) === RUN TestRaft_SnapshotRestore_PeerChange raft_test.go:1026: Disabled unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot raft_test.go:1132: Disabled unreliable test --- SKIP: TestRaft_AutoSnapshot (0.00s) === RUN TestRaft_UserSnapshot testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.077Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: entering follower state: follower="Node at 6af65f82-5092-4322-d891-3af4a67edd70 [Follower]" leader= testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.078Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c2ba2ed-a461-73c3-906c-9e95d6b47114 Address:6c2ba2ed-a461-73c3-906c-9e95d6b47114}]" testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.078Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c2ba2ed-a461-73c3-906c-9e95d6b47114 Address:6c2ba2ed-a461-73c3-906c-9e95d6b47114}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:11.078Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:11.078Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.078Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: entering follower state: follower="Node at 6c2ba2ed-a461-73c3-906c-9e95d6b47114 [Follower]" leader= testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.078Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: entering follower state: follower="Node at 6c2ba2ed-a461-73c3-906c-9e95d6b47114 [Follower]" leader= testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.145Z [WARN] server-6af65f82-5092-4322-d891-3af4a67edd70: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.146Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: entering candidate state: node="Node at 6af65f82-5092-4322-d891-3af4a67edd70 [Candidate]" term=3 testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.146Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: election won: tally=1 testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.146Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: entering leader state: leader="Node at 6af65f82-5092-4322-d891-3af4a67edd70 [Leader]" testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.148Z [WARN] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.148Z [WARN] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.148Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: entering candidate state: node="Node at 6c2ba2ed-a461-73c3-906c-9e95d6b47114 [Candidate]" term=2 testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.148Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: entering candidate state: node="Node at 6c2ba2ed-a461-73c3-906c-9e95d6b47114 [Candidate]" term=2 testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.148Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: election won: tally=1 testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.148Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: election won: tally=1 testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.148Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: entering leader state: leader="Node at 6c2ba2ed-a461-73c3-906c-9e95d6b47114 [Leader]" testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.148Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: entering leader state: leader="Node at 6c2ba2ed-a461-73c3-906c-9e95d6b47114 [Leader]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:11.254Z [INFO] stable state for Leader reached at 2021-08-25 09:28:11.148721677 +0000 UTC m=+10.912937566 (1 nodes), 70.164119ms from start of poll, 70.70003ms from cluster start. Timeout at 2021-08-25 09:28:11.253905261 +0000 UTC m=+11.018121230, 105.183664ms after stability testing.go:149: cluster: 2021-08-25T09:28:11.254Z [INFO] stable state for Leader reached at 2021-08-25 09:28:11.148721677 +0000 UTC m=+10.912937566 (1 nodes), 70.164119ms from start of poll, 70.70003ms from cluster start. Timeout at 2021-08-25 09:28:11.253905261 +0000 UTC m=+11.018121230, 105.183664ms after stability testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.254Z [ERROR] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.254Z [ERROR] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.254Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: starting snapshot up to: index=12 testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.254Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: starting snapshot up to: index=12 testing.go:153: 2021-08-25T09:28:11.254Z [INFO] creating new snapshot: path=/tmp/raft077058593/snapshots/2-12-1629883691254.tmp testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.272Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: compacting logs: from=1 to=2 testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.272Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: compacting logs: from=1 to=2 testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.272Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: snapshot complete up to: index=12 testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.272Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.20s) === RUN TestRaft_UserRestore testLoggerAdapter verbose: server-70513784-006e-26f1-77a2-2e72c552214b: 2021-08-25T09:28:11.275Z [INFO] server-70513784-006e-26f1-77a2-2e72c552214b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70513784-006e-26f1-77a2-2e72c552214b Address:70513784-006e-26f1-77a2-2e72c552214b} {Suffrage:Voter ID:server-ca0d195f-5484-93ff-72b6-e56b017a5c13 Address:ca0d195f-5484-93ff-72b6-e56b017a5c13} {Suffrage:Voter ID:server-637c2c1c-77d3-88aa-5912-743728615735 Address:637c2c1c-77d3-88aa-5912-743728615735}]" testing.go:149: server-70513784-006e-26f1-77a2-2e72c552214b: 2021-08-25T09:28:11.275Z [INFO] server-70513784-006e-26f1-77a2-2e72c552214b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70513784-006e-26f1-77a2-2e72c552214b Address:70513784-006e-26f1-77a2-2e72c552214b} {Suffrage:Voter ID:server-ca0d195f-5484-93ff-72b6-e56b017a5c13 Address:ca0d195f-5484-93ff-72b6-e56b017a5c13} {Suffrage:Voter ID:server-637c2c1c-77d3-88aa-5912-743728615735 Address:637c2c1c-77d3-88aa-5912-743728615735}]" testLoggerAdapter verbose: server-70513784-006e-26f1-77a2-2e72c552214b: 2021-08-25T09:28:11.275Z [INFO] server-70513784-006e-26f1-77a2-2e72c552214b: entering follower state: follower="Node at 70513784-006e-26f1-77a2-2e72c552214b [Follower]" leader= testing.go:149: server-70513784-006e-26f1-77a2-2e72c552214b: 2021-08-25T09:28:11.275Z [INFO] server-70513784-006e-26f1-77a2-2e72c552214b: entering follower state: follower="Node at 70513784-006e-26f1-77a2-2e72c552214b [Follower]" leader= testLoggerAdapter verbose: server-ca0d195f-5484-93ff-72b6-e56b017a5c13: 2021-08-25T09:28:11.275Z [INFO] server-ca0d195f-5484-93ff-72b6-e56b017a5c13: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70513784-006e-26f1-77a2-2e72c552214b Address:70513784-006e-26f1-77a2-2e72c552214b} {Suffrage:Voter ID:server-ca0d195f-5484-93ff-72b6-e56b017a5c13 Address:ca0d195f-5484-93ff-72b6-e56b017a5c13} {Suffrage:Voter ID:server-637c2c1c-77d3-88aa-5912-743728615735 Address:637c2c1c-77d3-88aa-5912-743728615735}]" testing.go:149: server-ca0d195f-5484-93ff-72b6-e56b017a5c13: 2021-08-25T09:28:11.275Z [INFO] server-ca0d195f-5484-93ff-72b6-e56b017a5c13: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70513784-006e-26f1-77a2-2e72c552214b Address:70513784-006e-26f1-77a2-2e72c552214b} {Suffrage:Voter ID:server-ca0d195f-5484-93ff-72b6-e56b017a5c13 Address:ca0d195f-5484-93ff-72b6-e56b017a5c13} {Suffrage:Voter ID:server-637c2c1c-77d3-88aa-5912-743728615735 Address:637c2c1c-77d3-88aa-5912-743728615735}]" testLoggerAdapter verbose: server-ca0d195f-5484-93ff-72b6-e56b017a5c13: 2021-08-25T09:28:11.276Z [INFO] server-ca0d195f-5484-93ff-72b6-e56b017a5c13: entering follower state: follower="Node at ca0d195f-5484-93ff-72b6-e56b017a5c13 [Follower]" leader= testing.go:149: server-ca0d195f-5484-93ff-72b6-e56b017a5c13: 2021-08-25T09:28:11.276Z [INFO] server-ca0d195f-5484-93ff-72b6-e56b017a5c13: entering follower state: follower="Node at ca0d195f-5484-93ff-72b6-e56b017a5c13 [Follower]" leader= testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.276Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70513784-006e-26f1-77a2-2e72c552214b Address:70513784-006e-26f1-77a2-2e72c552214b} {Suffrage:Voter ID:server-ca0d195f-5484-93ff-72b6-e56b017a5c13 Address:ca0d195f-5484-93ff-72b6-e56b017a5c13} {Suffrage:Voter ID:server-637c2c1c-77d3-88aa-5912-743728615735 Address:637c2c1c-77d3-88aa-5912-743728615735}]" testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.276Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70513784-006e-26f1-77a2-2e72c552214b Address:70513784-006e-26f1-77a2-2e72c552214b} {Suffrage:Voter ID:server-ca0d195f-5484-93ff-72b6-e56b017a5c13 Address:ca0d195f-5484-93ff-72b6-e56b017a5c13} {Suffrage:Voter ID:server-637c2c1c-77d3-88aa-5912-743728615735 Address:637c2c1c-77d3-88aa-5912-743728615735}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:11.276Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:11.276Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.276Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: entering follower state: follower="Node at 637c2c1c-77d3-88aa-5912-743728615735 [Follower]" leader= testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.276Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: entering follower state: follower="Node at 637c2c1c-77d3-88aa-5912-743728615735 [Follower]" leader= testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [WARN] server-637c2c1c-77d3-88aa-5912-743728615735: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [WARN] server-637c2c1c-77d3-88aa-5912-743728615735: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: entering candidate state: node="Node at 637c2c1c-77d3-88aa-5912-743728615735 [Candidate]" term=2 testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: entering candidate state: node="Node at 637c2c1c-77d3-88aa-5912-743728615735 [Candidate]" term=2 testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: election won: tally=2 testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: election won: tally=2 testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: entering leader state: leader="Node at 637c2c1c-77d3-88aa-5912-743728615735 [Leader]" testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: entering leader state: leader="Node at 637c2c1c-77d3-88aa-5912-743728615735 [Leader]" testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: added peer, starting replication: peer=server-70513784-006e-26f1-77a2-2e72c552214b testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: added peer, starting replication: peer=server-70513784-006e-26f1-77a2-2e72c552214b testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.329Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: added peer, starting replication: peer=server-ca0d195f-5484-93ff-72b6-e56b017a5c13 testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.329Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: added peer, starting replication: peer=server-ca0d195f-5484-93ff-72b6-e56b017a5c13 testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.329Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: pipelining replication: peer="{Voter server-70513784-006e-26f1-77a2-2e72c552214b 70513784-006e-26f1-77a2-2e72c552214b}" testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.329Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: pipelining replication: peer="{Voter server-70513784-006e-26f1-77a2-2e72c552214b 70513784-006e-26f1-77a2-2e72c552214b}" testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.329Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: pipelining replication: peer="{Voter server-ca0d195f-5484-93ff-72b6-e56b017a5c13 ca0d195f-5484-93ff-72b6-e56b017a5c13}" testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.329Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: pipelining replication: peer="{Voter server-ca0d195f-5484-93ff-72b6-e56b017a5c13 ca0d195f-5484-93ff-72b6-e56b017a5c13}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:11.434Z [INFO] stable state for Leader reached at 2021-08-25 09:28:11.329090293 +0000 UTC m=+11.093306182 (1 nodes), 52.562635ms from start of poll, 53.819078ms from cluster start. Timeout at 2021-08-25 09:28:11.434119592 +0000 UTC m=+11.198335561, 105.029379ms after stability testing.go:149: cluster: 2021-08-25T09:28:11.434Z [INFO] stable state for Leader reached at 2021-08-25 09:28:11.329090293 +0000 UTC m=+11.093306182 (1 nodes), 52.562635ms from start of poll, 53.819078ms from cluster start. Timeout at 2021-08-25 09:28:11.434119592 +0000 UTC m=+11.198335561, 105.029379ms after stability testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.434Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: starting snapshot up to: index=12 testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.434Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: starting snapshot up to: index=12 testing.go:153: 2021-08-25T09:28:11.434Z [INFO] creating new snapshot: path=/tmp/raft415421535/snapshots/2-12-1629883691434.tmp testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.453Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: snapshot complete up to: index=12 testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.453Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: snapshot complete up to: index=12 testing.go:153: 2021-08-25T09:28:11.454Z [INFO] creating new snapshot: path=/tmp/raft415421535/snapshots/2-23-1629883691454.tmp testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.472Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: copied to local snapshot: bytes=71 testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.472Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.472Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: restored user snapshot: index=1 testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.472Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: restored user snapshot: index=1 testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.472Z [ERROR] server-637c2c1c-77d3-88aa-5912-743728615735: failed to get log: index=23 error="log not found" testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.472Z [ERROR] server-637c2c1c-77d3-88aa-5912-743728615735: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.473Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: aborting pipeline replication: peer="{Voter server-ca0d195f-5484-93ff-72b6-e56b017a5c13 ca0d195f-5484-93ff-72b6-e56b017a5c13}" testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.473Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: aborting pipeline replication: peer="{Voter server-ca0d195f-5484-93ff-72b6-e56b017a5c13 ca0d195f-5484-93ff-72b6-e56b017a5c13}" testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.473Z [ERROR] server-637c2c1c-77d3-88aa-5912-743728615735: failed to get log: index=23 error="log not found" testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.473Z [ERROR] server-637c2c1c-77d3-88aa-5912-743728615735: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.473Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: aborting pipeline replication: peer="{Voter server-70513784-006e-26f1-77a2-2e72c552214b 70513784-006e-26f1-77a2-2e72c552214b}" testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.473Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: aborting pipeline replication: peer="{Voter server-70513784-006e-26f1-77a2-2e72c552214b 70513784-006e-26f1-77a2-2e72c552214b}" testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.479Z [ERROR] server-637c2c1c-77d3-88aa-5912-743728615735: failed to get log: index=23 error="log not found" testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.479Z [ERROR] server-637c2c1c-77d3-88aa-5912-743728615735: failed to get log: index=23 error="log not found" testing.go:153: 2021-08-25T09:28:11.480Z [INFO] creating new snapshot: path=/tmp/raft144830203/snapshots/2-23-1629883691479.tmp testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.482Z [ERROR] server-637c2c1c-77d3-88aa-5912-743728615735: failed to get log: index=23 error="log not found" testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.482Z [ERROR] server-637c2c1c-77d3-88aa-5912-743728615735: failed to get log: index=23 error="log not found" testing.go:153: 2021-08-25T09:28:11.483Z [INFO] creating new snapshot: path=/tmp/raft803796197/snapshots/2-23-1629883691483.tmp testLoggerAdapter verbose: server-70513784-006e-26f1-77a2-2e72c552214b: 2021-08-25T09:28:11.508Z [INFO] server-70513784-006e-26f1-77a2-2e72c552214b: copied to local snapshot: bytes=71 testing.go:149: server-70513784-006e-26f1-77a2-2e72c552214b: 2021-08-25T09:28:11.508Z [INFO] server-70513784-006e-26f1-77a2-2e72c552214b: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-70513784-006e-26f1-77a2-2e72c552214b: 2021-08-25T09:28:11.509Z [INFO] server-70513784-006e-26f1-77a2-2e72c552214b: Installed remote snapshot testing.go:149: server-70513784-006e-26f1-77a2-2e72c552214b: 2021-08-25T09:28:11.509Z [INFO] server-70513784-006e-26f1-77a2-2e72c552214b: Installed remote snapshot testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.509Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: pipelining replication: peer="{Voter server-70513784-006e-26f1-77a2-2e72c552214b 70513784-006e-26f1-77a2-2e72c552214b}" testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.509Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: pipelining replication: peer="{Voter server-70513784-006e-26f1-77a2-2e72c552214b 70513784-006e-26f1-77a2-2e72c552214b}" testLoggerAdapter verbose: server-ca0d195f-5484-93ff-72b6-e56b017a5c13: 2021-08-25T09:28:11.512Z [INFO] server-ca0d195f-5484-93ff-72b6-e56b017a5c13: copied to local snapshot: bytes=71 testing.go:149: server-ca0d195f-5484-93ff-72b6-e56b017a5c13: 2021-08-25T09:28:11.512Z [INFO] server-ca0d195f-5484-93ff-72b6-e56b017a5c13: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-ca0d195f-5484-93ff-72b6-e56b017a5c13: 2021-08-25T09:28:11.512Z [INFO] server-ca0d195f-5484-93ff-72b6-e56b017a5c13: Installed remote snapshot testing.go:149: server-ca0d195f-5484-93ff-72b6-e56b017a5c13: 2021-08-25T09:28:11.512Z [INFO] server-ca0d195f-5484-93ff-72b6-e56b017a5c13: Installed remote snapshot testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.513Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: pipelining replication: peer="{Voter server-ca0d195f-5484-93ff-72b6-e56b017a5c13 ca0d195f-5484-93ff-72b6-e56b017a5c13}" testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.513Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: pipelining replication: peer="{Voter server-ca0d195f-5484-93ff-72b6-e56b017a5c13 ca0d195f-5484-93ff-72b6-e56b017a5c13}" testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.526Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: aborting pipeline replication: peer="{Voter server-70513784-006e-26f1-77a2-2e72c552214b 70513784-006e-26f1-77a2-2e72c552214b}" testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.526Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: aborting pipeline replication: peer="{Voter server-70513784-006e-26f1-77a2-2e72c552214b 70513784-006e-26f1-77a2-2e72c552214b}" testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.526Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: aborting pipeline replication: peer="{Voter server-ca0d195f-5484-93ff-72b6-e56b017a5c13 ca0d195f-5484-93ff-72b6-e56b017a5c13}" testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.526Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: aborting pipeline replication: peer="{Voter server-ca0d195f-5484-93ff-72b6-e56b017a5c13 ca0d195f-5484-93ff-72b6-e56b017a5c13}" testLoggerAdapter verbose: server-f49ced75-b740-fc85-917e-df19e9bc02f4: 2021-08-25T09:28:11.530Z [INFO] server-f49ced75-b740-fc85-917e-df19e9bc02f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f49ced75-b740-fc85-917e-df19e9bc02f4 Address:f49ced75-b740-fc85-917e-df19e9bc02f4} {Suffrage:Voter ID:server-567bc5cd-c47d-652e-4739-b45ffa36f0d2 Address:567bc5cd-c47d-652e-4739-b45ffa36f0d2} {Suffrage:Voter ID:server-28b1b5fb-9683-9abb-066a-ab7c417f8edc Address:28b1b5fb-9683-9abb-066a-ab7c417f8edc}]" testing.go:149: server-f49ced75-b740-fc85-917e-df19e9bc02f4: 2021-08-25T09:28:11.530Z [INFO] server-f49ced75-b740-fc85-917e-df19e9bc02f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f49ced75-b740-fc85-917e-df19e9bc02f4 Address:f49ced75-b740-fc85-917e-df19e9bc02f4} {Suffrage:Voter ID:server-567bc5cd-c47d-652e-4739-b45ffa36f0d2 Address:567bc5cd-c47d-652e-4739-b45ffa36f0d2} {Suffrage:Voter ID:server-28b1b5fb-9683-9abb-066a-ab7c417f8edc Address:28b1b5fb-9683-9abb-066a-ab7c417f8edc}]" testLoggerAdapter verbose: server-f49ced75-b740-fc85-917e-df19e9bc02f4: 2021-08-25T09:28:11.531Z [INFO] server-f49ced75-b740-fc85-917e-df19e9bc02f4: entering follower state: follower="Node at f49ced75-b740-fc85-917e-df19e9bc02f4 [Follower]" leader= testing.go:149: server-f49ced75-b740-fc85-917e-df19e9bc02f4: 2021-08-25T09:28:11.531Z [INFO] server-f49ced75-b740-fc85-917e-df19e9bc02f4: entering follower state: follower="Node at f49ced75-b740-fc85-917e-df19e9bc02f4 [Follower]" leader= testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.531Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f49ced75-b740-fc85-917e-df19e9bc02f4 Address:f49ced75-b740-fc85-917e-df19e9bc02f4} {Suffrage:Voter ID:server-567bc5cd-c47d-652e-4739-b45ffa36f0d2 Address:567bc5cd-c47d-652e-4739-b45ffa36f0d2} {Suffrage:Voter ID:server-28b1b5fb-9683-9abb-066a-ab7c417f8edc Address:28b1b5fb-9683-9abb-066a-ab7c417f8edc}]" testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.531Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f49ced75-b740-fc85-917e-df19e9bc02f4 Address:f49ced75-b740-fc85-917e-df19e9bc02f4} {Suffrage:Voter ID:server-567bc5cd-c47d-652e-4739-b45ffa36f0d2 Address:567bc5cd-c47d-652e-4739-b45ffa36f0d2} {Suffrage:Voter ID:server-28b1b5fb-9683-9abb-066a-ab7c417f8edc Address:28b1b5fb-9683-9abb-066a-ab7c417f8edc}]" testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.531Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: entering follower state: follower="Node at 567bc5cd-c47d-652e-4739-b45ffa36f0d2 [Follower]" leader= testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.531Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: entering follower state: follower="Node at 567bc5cd-c47d-652e-4739-b45ffa36f0d2 [Follower]" leader= testLoggerAdapter verbose: server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: 2021-08-25T09:28:11.531Z [INFO] server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f49ced75-b740-fc85-917e-df19e9bc02f4 Address:f49ced75-b740-fc85-917e-df19e9bc02f4} {Suffrage:Voter ID:server-567bc5cd-c47d-652e-4739-b45ffa36f0d2 Address:567bc5cd-c47d-652e-4739-b45ffa36f0d2} {Suffrage:Voter ID:server-28b1b5fb-9683-9abb-066a-ab7c417f8edc Address:28b1b5fb-9683-9abb-066a-ab7c417f8edc}]" testing.go:149: server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: 2021-08-25T09:28:11.531Z [INFO] server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f49ced75-b740-fc85-917e-df19e9bc02f4 Address:f49ced75-b740-fc85-917e-df19e9bc02f4} {Suffrage:Voter ID:server-567bc5cd-c47d-652e-4739-b45ffa36f0d2 Address:567bc5cd-c47d-652e-4739-b45ffa36f0d2} {Suffrage:Voter ID:server-28b1b5fb-9683-9abb-066a-ab7c417f8edc Address:28b1b5fb-9683-9abb-066a-ab7c417f8edc}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:11.531Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:11.531Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: 2021-08-25T09:28:11.531Z [INFO] server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: entering follower state: follower="Node at 28b1b5fb-9683-9abb-066a-ab7c417f8edc [Follower]" leader= testing.go:149: server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: 2021-08-25T09:28:11.531Z [INFO] server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: entering follower state: follower="Node at 28b1b5fb-9683-9abb-066a-ab7c417f8edc [Follower]" leader= testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [WARN] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [WARN] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: entering candidate state: node="Node at 567bc5cd-c47d-652e-4739-b45ffa36f0d2 [Candidate]" term=2 testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: entering candidate state: node="Node at 567bc5cd-c47d-652e-4739-b45ffa36f0d2 [Candidate]" term=2 testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: election won: tally=2 testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: election won: tally=2 testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: entering leader state: leader="Node at 567bc5cd-c47d-652e-4739-b45ffa36f0d2 [Leader]" testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: entering leader state: leader="Node at 567bc5cd-c47d-652e-4739-b45ffa36f0d2 [Leader]" testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: added peer, starting replication: peer=server-f49ced75-b740-fc85-917e-df19e9bc02f4 testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: added peer, starting replication: peer=server-f49ced75-b740-fc85-917e-df19e9bc02f4 testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: added peer, starting replication: peer=server-28b1b5fb-9683-9abb-066a-ab7c417f8edc testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: added peer, starting replication: peer=server-28b1b5fb-9683-9abb-066a-ab7c417f8edc testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: pipelining replication: peer="{Voter server-f49ced75-b740-fc85-917e-df19e9bc02f4 f49ced75-b740-fc85-917e-df19e9bc02f4}" testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: pipelining replication: peer="{Voter server-f49ced75-b740-fc85-917e-df19e9bc02f4 f49ced75-b740-fc85-917e-df19e9bc02f4}" testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.607Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: pipelining replication: peer="{Voter server-28b1b5fb-9683-9abb-066a-ab7c417f8edc 28b1b5fb-9683-9abb-066a-ab7c417f8edc}" testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.607Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: pipelining replication: peer="{Voter server-28b1b5fb-9683-9abb-066a-ab7c417f8edc 28b1b5fb-9683-9abb-066a-ab7c417f8edc}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:11.712Z [INFO] stable state for Leader reached at 2021-08-25 09:28:11.606893934 +0000 UTC m=+11.371109823 (1 nodes), 75.161714ms from start of poll, 76.211277ms from cluster start. Timeout at 2021-08-25 09:28:11.712006065 +0000 UTC m=+11.476222034, 105.112211ms after stability testing.go:149: cluster: 2021-08-25T09:28:11.712Z [INFO] stable state for Leader reached at 2021-08-25 09:28:11.606893934 +0000 UTC m=+11.371109823 (1 nodes), 75.161714ms from start of poll, 76.211277ms from cluster start. Timeout at 2021-08-25 09:28:11.712006065 +0000 UTC m=+11.476222034, 105.112211ms after stability testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.712Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: starting snapshot up to: index=12 testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.712Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: starting snapshot up to: index=12 testing.go:153: 2021-08-25T09:28:11.712Z [INFO] creating new snapshot: path=/tmp/raft920409603/snapshots/2-12-1629883691712.tmp testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.733Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: snapshot complete up to: index=12 testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.733Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: snapshot complete up to: index=12 testing.go:153: 2021-08-25T09:28:11.734Z [INFO] creating new snapshot: path=/tmp/raft920409603/snapshots/2-23-1629883691734.tmp testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.751Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: copied to local snapshot: bytes=71 testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.751Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.751Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: restored user snapshot: index=1 testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.751Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: restored user snapshot: index=1 testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.752Z [ERROR] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: failed to get log: index=23 error="log not found" testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.752Z [ERROR] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.752Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: aborting pipeline replication: peer="{Voter server-28b1b5fb-9683-9abb-066a-ab7c417f8edc 28b1b5fb-9683-9abb-066a-ab7c417f8edc}" testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.752Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: aborting pipeline replication: peer="{Voter server-28b1b5fb-9683-9abb-066a-ab7c417f8edc 28b1b5fb-9683-9abb-066a-ab7c417f8edc}" testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.752Z [ERROR] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: failed to get log: index=23 error="log not found" testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.752Z [ERROR] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.752Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: aborting pipeline replication: peer="{Voter server-f49ced75-b740-fc85-917e-df19e9bc02f4 f49ced75-b740-fc85-917e-df19e9bc02f4}" testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.752Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: aborting pipeline replication: peer="{Voter server-f49ced75-b740-fc85-917e-df19e9bc02f4 f49ced75-b740-fc85-917e-df19e9bc02f4}" testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.757Z [ERROR] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: failed to get log: index=23 error="log not found" testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.757Z [ERROR] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: failed to get log: index=23 error="log not found" testing.go:153: 2021-08-25T09:28:11.758Z [INFO] creating new snapshot: path=/tmp/raft718466025/snapshots/2-23-1629883691758.tmp testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.764Z [ERROR] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: failed to get log: index=23 error="log not found" testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.764Z [ERROR] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: failed to get log: index=23 error="log not found" testing.go:153: 2021-08-25T09:28:11.765Z [INFO] creating new snapshot: path=/tmp/raft277194541/snapshots/2-23-1629883691765.tmp testLoggerAdapter verbose: server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: 2021-08-25T09:28:11.784Z [INFO] server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: copied to local snapshot: bytes=71 testing.go:149: server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: 2021-08-25T09:28:11.784Z [INFO] server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: 2021-08-25T09:28:11.784Z [INFO] server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: Installed remote snapshot testing.go:149: server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: 2021-08-25T09:28:11.784Z [INFO] server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: Installed remote snapshot testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.784Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: pipelining replication: peer="{Voter server-28b1b5fb-9683-9abb-066a-ab7c417f8edc 28b1b5fb-9683-9abb-066a-ab7c417f8edc}" testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.784Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: pipelining replication: peer="{Voter server-28b1b5fb-9683-9abb-066a-ab7c417f8edc 28b1b5fb-9683-9abb-066a-ab7c417f8edc}" testLoggerAdapter verbose: server-f49ced75-b740-fc85-917e-df19e9bc02f4: 2021-08-25T09:28:11.785Z [INFO] server-f49ced75-b740-fc85-917e-df19e9bc02f4: copied to local snapshot: bytes=71 testing.go:149: server-f49ced75-b740-fc85-917e-df19e9bc02f4: 2021-08-25T09:28:11.785Z [INFO] server-f49ced75-b740-fc85-917e-df19e9bc02f4: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-f49ced75-b740-fc85-917e-df19e9bc02f4: 2021-08-25T09:28:11.785Z [INFO] server-f49ced75-b740-fc85-917e-df19e9bc02f4: Installed remote snapshot testing.go:149: server-f49ced75-b740-fc85-917e-df19e9bc02f4: 2021-08-25T09:28:11.785Z [INFO] server-f49ced75-b740-fc85-917e-df19e9bc02f4: Installed remote snapshot testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.785Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: pipelining replication: peer="{Voter server-f49ced75-b740-fc85-917e-df19e9bc02f4 f49ced75-b740-fc85-917e-df19e9bc02f4}" testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.785Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: pipelining replication: peer="{Voter server-f49ced75-b740-fc85-917e-df19e9bc02f4 f49ced75-b740-fc85-917e-df19e9bc02f4}" testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.801Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: aborting pipeline replication: peer="{Voter server-28b1b5fb-9683-9abb-066a-ab7c417f8edc 28b1b5fb-9683-9abb-066a-ab7c417f8edc}" testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.801Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: aborting pipeline replication: peer="{Voter server-28b1b5fb-9683-9abb-066a-ab7c417f8edc 28b1b5fb-9683-9abb-066a-ab7c417f8edc}" testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.801Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: aborting pipeline replication: peer="{Voter server-f49ced75-b740-fc85-917e-df19e9bc02f4 f49ced75-b740-fc85-917e-df19e9bc02f4}" testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.801Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: aborting pipeline replication: peer="{Voter server-f49ced75-b740-fc85-917e-df19e9bc02f4 f49ced75-b740-fc85-917e-df19e9bc02f4}" testLoggerAdapter verbose: server-af544a6b-77d8-2d43-79d5-72c50e5769a4: 2021-08-25T09:28:11.805Z [INFO] server-af544a6b-77d8-2d43-79d5-72c50e5769a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af544a6b-77d8-2d43-79d5-72c50e5769a4 Address:af544a6b-77d8-2d43-79d5-72c50e5769a4} {Suffrage:Voter ID:server-35dd04bf-32c2-aa68-0421-516c39020c2d Address:35dd04bf-32c2-aa68-0421-516c39020c2d} {Suffrage:Voter ID:server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc Address:f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}]" testing.go:149: server-af544a6b-77d8-2d43-79d5-72c50e5769a4: 2021-08-25T09:28:11.805Z [INFO] server-af544a6b-77d8-2d43-79d5-72c50e5769a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af544a6b-77d8-2d43-79d5-72c50e5769a4 Address:af544a6b-77d8-2d43-79d5-72c50e5769a4} {Suffrage:Voter ID:server-35dd04bf-32c2-aa68-0421-516c39020c2d Address:35dd04bf-32c2-aa68-0421-516c39020c2d} {Suffrage:Voter ID:server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc Address:f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}]" testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.805Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af544a6b-77d8-2d43-79d5-72c50e5769a4 Address:af544a6b-77d8-2d43-79d5-72c50e5769a4} {Suffrage:Voter ID:server-35dd04bf-32c2-aa68-0421-516c39020c2d Address:35dd04bf-32c2-aa68-0421-516c39020c2d} {Suffrage:Voter ID:server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc Address:f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}]" testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.805Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af544a6b-77d8-2d43-79d5-72c50e5769a4 Address:af544a6b-77d8-2d43-79d5-72c50e5769a4} {Suffrage:Voter ID:server-35dd04bf-32c2-aa68-0421-516c39020c2d Address:35dd04bf-32c2-aa68-0421-516c39020c2d} {Suffrage:Voter ID:server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc Address:f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}]" testLoggerAdapter verbose: server-af544a6b-77d8-2d43-79d5-72c50e5769a4: 2021-08-25T09:28:11.806Z [INFO] server-af544a6b-77d8-2d43-79d5-72c50e5769a4: entering follower state: follower="Node at af544a6b-77d8-2d43-79d5-72c50e5769a4 [Follower]" leader= testLoggerAdapter verbose: server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: 2021-08-25T09:28:11.806Z [INFO] server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af544a6b-77d8-2d43-79d5-72c50e5769a4 Address:af544a6b-77d8-2d43-79d5-72c50e5769a4} {Suffrage:Voter ID:server-35dd04bf-32c2-aa68-0421-516c39020c2d Address:35dd04bf-32c2-aa68-0421-516c39020c2d} {Suffrage:Voter ID:server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc Address:f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}]" testing.go:149: server-af544a6b-77d8-2d43-79d5-72c50e5769a4: 2021-08-25T09:28:11.806Z [INFO] server-af544a6b-77d8-2d43-79d5-72c50e5769a4: entering follower state: follower="Node at af544a6b-77d8-2d43-79d5-72c50e5769a4 [Follower]" leader= testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.806Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: entering follower state: follower="Node at 35dd04bf-32c2-aa68-0421-516c39020c2d [Follower]" leader= testing.go:149: server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: 2021-08-25T09:28:11.806Z [INFO] server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af544a6b-77d8-2d43-79d5-72c50e5769a4 Address:af544a6b-77d8-2d43-79d5-72c50e5769a4} {Suffrage:Voter ID:server-35dd04bf-32c2-aa68-0421-516c39020c2d Address:35dd04bf-32c2-aa68-0421-516c39020c2d} {Suffrage:Voter ID:server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc Address:f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:11.806Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:11.806Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.806Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: entering follower state: follower="Node at 35dd04bf-32c2-aa68-0421-516c39020c2d [Follower]" leader= testLoggerAdapter verbose: server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: 2021-08-25T09:28:11.806Z [INFO] server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: entering follower state: follower="Node at f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc [Follower]" leader= testing.go:149: server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: 2021-08-25T09:28:11.806Z [INFO] server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: entering follower state: follower="Node at f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc [Follower]" leader= testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [WARN] server-35dd04bf-32c2-aa68-0421-516c39020c2d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [WARN] server-35dd04bf-32c2-aa68-0421-516c39020c2d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: entering candidate state: node="Node at 35dd04bf-32c2-aa68-0421-516c39020c2d [Candidate]" term=2 testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: entering candidate state: node="Node at 35dd04bf-32c2-aa68-0421-516c39020c2d [Candidate]" term=2 testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: election won: tally=2 testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: election won: tally=2 testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: entering leader state: leader="Node at 35dd04bf-32c2-aa68-0421-516c39020c2d [Leader]" testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: entering leader state: leader="Node at 35dd04bf-32c2-aa68-0421-516c39020c2d [Leader]" testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: added peer, starting replication: peer=server-af544a6b-77d8-2d43-79d5-72c50e5769a4 testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: added peer, starting replication: peer=server-af544a6b-77d8-2d43-79d5-72c50e5769a4 testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.872Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: added peer, starting replication: peer=server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.872Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: added peer, starting replication: peer=server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.872Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: pipelining replication: peer="{Voter server-af544a6b-77d8-2d43-79d5-72c50e5769a4 af544a6b-77d8-2d43-79d5-72c50e5769a4}" testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.872Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: pipelining replication: peer="{Voter server-af544a6b-77d8-2d43-79d5-72c50e5769a4 af544a6b-77d8-2d43-79d5-72c50e5769a4}" testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.872Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: pipelining replication: peer="{Voter server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}" testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.872Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: pipelining replication: peer="{Voter server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:11.977Z [INFO] stable state for Leader reached at 2021-08-25 09:28:11.872157961 +0000 UTC m=+11.636373850 (1 nodes), 65.657658ms from start of poll, 66.778714ms from cluster start. Timeout at 2021-08-25 09:28:11.977166021 +0000 UTC m=+11.741381990, 105.00814ms after stability testing.go:149: cluster: 2021-08-25T09:28:11.977Z [INFO] stable state for Leader reached at 2021-08-25 09:28:11.872157961 +0000 UTC m=+11.636373850 (1 nodes), 65.657658ms from start of poll, 66.778714ms from cluster start. Timeout at 2021-08-25 09:28:11.977166021 +0000 UTC m=+11.741381990, 105.00814ms after stability testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.977Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: starting snapshot up to: index=12 testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.977Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: starting snapshot up to: index=12 testing.go:153: 2021-08-25T09:28:11.978Z [INFO] creating new snapshot: path=/tmp/raft104314033/snapshots/2-12-1629883691978.tmp testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.997Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: snapshot complete up to: index=12 testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.997Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: snapshot complete up to: index=12 testing.go:153: 2021-08-25T09:28:11.998Z [INFO] creating new snapshot: path=/tmp/raft104314033/snapshots/2-23-1629883691998.tmp testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.022Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: copied to local snapshot: bytes=71 testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.022Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.023Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: restored user snapshot: index=1 testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.023Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: restored user snapshot: index=1 testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.023Z [ERROR] server-35dd04bf-32c2-aa68-0421-516c39020c2d: failed to get log: index=23 error="log not found" testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.023Z [ERROR] server-35dd04bf-32c2-aa68-0421-516c39020c2d: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.023Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: aborting pipeline replication: peer="{Voter server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}" testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.023Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: aborting pipeline replication: peer="{Voter server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}" testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.023Z [ERROR] server-35dd04bf-32c2-aa68-0421-516c39020c2d: failed to get log: index=23 error="log not found" testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.023Z [ERROR] server-35dd04bf-32c2-aa68-0421-516c39020c2d: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.024Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: aborting pipeline replication: peer="{Voter server-af544a6b-77d8-2d43-79d5-72c50e5769a4 af544a6b-77d8-2d43-79d5-72c50e5769a4}" testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.024Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: aborting pipeline replication: peer="{Voter server-af544a6b-77d8-2d43-79d5-72c50e5769a4 af544a6b-77d8-2d43-79d5-72c50e5769a4}" testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.030Z [ERROR] server-35dd04bf-32c2-aa68-0421-516c39020c2d: failed to get log: index=23 error="log not found" testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.030Z [ERROR] server-35dd04bf-32c2-aa68-0421-516c39020c2d: failed to get log: index=23 error="log not found" testing.go:153: 2021-08-25T09:28:12.031Z [INFO] creating new snapshot: path=/tmp/raft372438503/snapshots/2-23-1629883692031.tmp testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.033Z [ERROR] server-35dd04bf-32c2-aa68-0421-516c39020c2d: failed to get log: index=23 error="log not found" testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.033Z [ERROR] server-35dd04bf-32c2-aa68-0421-516c39020c2d: failed to get log: index=23 error="log not found" testing.go:153: 2021-08-25T09:28:12.034Z [INFO] creating new snapshot: path=/tmp/raft077980171/snapshots/2-23-1629883692034.tmp testLoggerAdapter verbose: server-af544a6b-77d8-2d43-79d5-72c50e5769a4: 2021-08-25T09:28:12.061Z [INFO] server-af544a6b-77d8-2d43-79d5-72c50e5769a4: copied to local snapshot: bytes=71 testing.go:149: server-af544a6b-77d8-2d43-79d5-72c50e5769a4: 2021-08-25T09:28:12.061Z [INFO] server-af544a6b-77d8-2d43-79d5-72c50e5769a4: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-af544a6b-77d8-2d43-79d5-72c50e5769a4: 2021-08-25T09:28:12.061Z [INFO] server-af544a6b-77d8-2d43-79d5-72c50e5769a4: Installed remote snapshot testing.go:149: server-af544a6b-77d8-2d43-79d5-72c50e5769a4: 2021-08-25T09:28:12.061Z [INFO] server-af544a6b-77d8-2d43-79d5-72c50e5769a4: Installed remote snapshot testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.062Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: pipelining replication: peer="{Voter server-af544a6b-77d8-2d43-79d5-72c50e5769a4 af544a6b-77d8-2d43-79d5-72c50e5769a4}" testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.062Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: pipelining replication: peer="{Voter server-af544a6b-77d8-2d43-79d5-72c50e5769a4 af544a6b-77d8-2d43-79d5-72c50e5769a4}" testLoggerAdapter verbose: server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: 2021-08-25T09:28:12.065Z [INFO] server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: copied to local snapshot: bytes=71 testing.go:149: server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: 2021-08-25T09:28:12.065Z [INFO] server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: 2021-08-25T09:28:12.066Z [INFO] server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: Installed remote snapshot testing.go:149: server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: 2021-08-25T09:28:12.066Z [INFO] server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: Installed remote snapshot testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.066Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: pipelining replication: peer="{Voter server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}" testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.066Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: pipelining replication: peer="{Voter server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}" testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.083Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: aborting pipeline replication: peer="{Voter server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}" testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.083Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: aborting pipeline replication: peer="{Voter server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}" testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.083Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: aborting pipeline replication: peer="{Voter server-af544a6b-77d8-2d43-79d5-72c50e5769a4 af544a6b-77d8-2d43-79d5-72c50e5769a4}" testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.083Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: aborting pipeline replication: peer="{Voter server-af544a6b-77d8-2d43-79d5-72c50e5769a4 af544a6b-77d8-2d43-79d5-72c50e5769a4}" testLoggerAdapter verbose: server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: 2021-08-25T09:28:12.089Z [INFO] server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b092822-0ccd-384f-79f0-bd4d6ddb280c Address:1b092822-0ccd-384f-79f0-bd4d6ddb280c} {Suffrage:Voter ID:server-fec26d77-9d21-8ed3-d991-338d6313021e Address:fec26d77-9d21-8ed3-d991-338d6313021e} {Suffrage:Voter ID:server-f8ea6912-c680-0e36-82fb-4ced5504d5ea Address:f8ea6912-c680-0e36-82fb-4ced5504d5ea}]" testing.go:149: server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: 2021-08-25T09:28:12.089Z [INFO] server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b092822-0ccd-384f-79f0-bd4d6ddb280c Address:1b092822-0ccd-384f-79f0-bd4d6ddb280c} {Suffrage:Voter ID:server-fec26d77-9d21-8ed3-d991-338d6313021e Address:fec26d77-9d21-8ed3-d991-338d6313021e} {Suffrage:Voter ID:server-f8ea6912-c680-0e36-82fb-4ced5504d5ea Address:f8ea6912-c680-0e36-82fb-4ced5504d5ea}]" testLoggerAdapter verbose: server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: 2021-08-25T09:28:12.089Z [INFO] server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: entering follower state: follower="Node at 1b092822-0ccd-384f-79f0-bd4d6ddb280c [Follower]" leader= testing.go:149: server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: 2021-08-25T09:28:12.089Z [INFO] server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: entering follower state: follower="Node at 1b092822-0ccd-384f-79f0-bd4d6ddb280c [Follower]" leader= testLoggerAdapter verbose: server-fec26d77-9d21-8ed3-d991-338d6313021e: 2021-08-25T09:28:12.089Z [INFO] server-fec26d77-9d21-8ed3-d991-338d6313021e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b092822-0ccd-384f-79f0-bd4d6ddb280c Address:1b092822-0ccd-384f-79f0-bd4d6ddb280c} {Suffrage:Voter ID:server-fec26d77-9d21-8ed3-d991-338d6313021e Address:fec26d77-9d21-8ed3-d991-338d6313021e} {Suffrage:Voter ID:server-f8ea6912-c680-0e36-82fb-4ced5504d5ea Address:f8ea6912-c680-0e36-82fb-4ced5504d5ea}]" testing.go:149: server-fec26d77-9d21-8ed3-d991-338d6313021e: 2021-08-25T09:28:12.089Z [INFO] server-fec26d77-9d21-8ed3-d991-338d6313021e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b092822-0ccd-384f-79f0-bd4d6ddb280c Address:1b092822-0ccd-384f-79f0-bd4d6ddb280c} {Suffrage:Voter ID:server-fec26d77-9d21-8ed3-d991-338d6313021e Address:fec26d77-9d21-8ed3-d991-338d6313021e} {Suffrage:Voter ID:server-f8ea6912-c680-0e36-82fb-4ced5504d5ea Address:f8ea6912-c680-0e36-82fb-4ced5504d5ea}]" testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.089Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b092822-0ccd-384f-79f0-bd4d6ddb280c Address:1b092822-0ccd-384f-79f0-bd4d6ddb280c} {Suffrage:Voter ID:server-fec26d77-9d21-8ed3-d991-338d6313021e Address:fec26d77-9d21-8ed3-d991-338d6313021e} {Suffrage:Voter ID:server-f8ea6912-c680-0e36-82fb-4ced5504d5ea Address:f8ea6912-c680-0e36-82fb-4ced5504d5ea}]" testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.089Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b092822-0ccd-384f-79f0-bd4d6ddb280c Address:1b092822-0ccd-384f-79f0-bd4d6ddb280c} {Suffrage:Voter ID:server-fec26d77-9d21-8ed3-d991-338d6313021e Address:fec26d77-9d21-8ed3-d991-338d6313021e} {Suffrage:Voter ID:server-f8ea6912-c680-0e36-82fb-4ced5504d5ea Address:f8ea6912-c680-0e36-82fb-4ced5504d5ea}]" testLoggerAdapter verbose: server-fec26d77-9d21-8ed3-d991-338d6313021e: 2021-08-25T09:28:12.089Z [INFO] server-fec26d77-9d21-8ed3-d991-338d6313021e: entering follower state: follower="Node at fec26d77-9d21-8ed3-d991-338d6313021e [Follower]" leader= testLoggerAdapter verbose: cluster: 2021-08-25T09:28:12.090Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-fec26d77-9d21-8ed3-d991-338d6313021e: 2021-08-25T09:28:12.089Z [INFO] server-fec26d77-9d21-8ed3-d991-338d6313021e: entering follower state: follower="Node at fec26d77-9d21-8ed3-d991-338d6313021e [Follower]" leader= testing.go:149: cluster: 2021-08-25T09:28:12.090Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.090Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: entering follower state: follower="Node at f8ea6912-c680-0e36-82fb-4ced5504d5ea [Follower]" leader= testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.090Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: entering follower state: follower="Node at f8ea6912-c680-0e36-82fb-4ced5504d5ea [Follower]" leader= testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.145Z [WARN] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.145Z [WARN] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.145Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: entering candidate state: node="Node at f8ea6912-c680-0e36-82fb-4ced5504d5ea [Candidate]" term=2 testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.145Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: entering candidate state: node="Node at f8ea6912-c680-0e36-82fb-4ced5504d5ea [Candidate]" term=2 testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: election won: tally=2 testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: election won: tally=2 testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: entering leader state: leader="Node at f8ea6912-c680-0e36-82fb-4ced5504d5ea [Leader]" testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: entering leader state: leader="Node at f8ea6912-c680-0e36-82fb-4ced5504d5ea [Leader]" testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: added peer, starting replication: peer=server-1b092822-0ccd-384f-79f0-bd4d6ddb280c testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: added peer, starting replication: peer=server-1b092822-0ccd-384f-79f0-bd4d6ddb280c testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: added peer, starting replication: peer=server-fec26d77-9d21-8ed3-d991-338d6313021e testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: added peer, starting replication: peer=server-fec26d77-9d21-8ed3-d991-338d6313021e testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: pipelining replication: peer="{Voter server-fec26d77-9d21-8ed3-d991-338d6313021e fec26d77-9d21-8ed3-d991-338d6313021e}" testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: pipelining replication: peer="{Voter server-fec26d77-9d21-8ed3-d991-338d6313021e fec26d77-9d21-8ed3-d991-338d6313021e}" testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: pipelining replication: peer="{Voter server-1b092822-0ccd-384f-79f0-bd4d6ddb280c 1b092822-0ccd-384f-79f0-bd4d6ddb280c}" testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: pipelining replication: peer="{Voter server-1b092822-0ccd-384f-79f0-bd4d6ddb280c 1b092822-0ccd-384f-79f0-bd4d6ddb280c}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:12.252Z [INFO] stable state for Leader reached at 2021-08-25 09:28:12.146928903 +0000 UTC m=+11.911144792 (1 nodes), 56.773983ms from start of poll, 58.113558ms from cluster start. Timeout at 2021-08-25 09:28:12.252042954 +0000 UTC m=+12.016258843, 105.114051ms after stability testing.go:149: cluster: 2021-08-25T09:28:12.252Z [INFO] stable state for Leader reached at 2021-08-25 09:28:12.146928903 +0000 UTC m=+11.911144792 (1 nodes), 56.773983ms from start of poll, 58.113558ms from cluster start. Timeout at 2021-08-25 09:28:12.252042954 +0000 UTC m=+12.016258843, 105.114051ms after stability testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.252Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: starting snapshot up to: index=12 testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.252Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: starting snapshot up to: index=12 testing.go:153: 2021-08-25T09:28:12.252Z [INFO] creating new snapshot: path=/tmp/raft294703737/snapshots/2-12-1629883692252.tmp testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.271Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: snapshot complete up to: index=12 testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.271Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: snapshot complete up to: index=12 testing.go:153: 2021-08-25T09:28:12.272Z [INFO] creating new snapshot: path=/tmp/raft294703737/snapshots/2-113-1629883692272.tmp testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.290Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: copied to local snapshot: bytes=71 testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.290Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: restored user snapshot: index=1 testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: restored user snapshot: index=1 testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [ERROR] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: failed to get log: index=23 error="log not found" testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [ERROR] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [ERROR] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: failed to get log: index=23 error="log not found" testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [ERROR] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: aborting pipeline replication: peer="{Voter server-fec26d77-9d21-8ed3-d991-338d6313021e fec26d77-9d21-8ed3-d991-338d6313021e}" testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: aborting pipeline replication: peer="{Voter server-fec26d77-9d21-8ed3-d991-338d6313021e fec26d77-9d21-8ed3-d991-338d6313021e}" testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: aborting pipeline replication: peer="{Voter server-1b092822-0ccd-384f-79f0-bd4d6ddb280c 1b092822-0ccd-384f-79f0-bd4d6ddb280c}" testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: aborting pipeline replication: peer="{Voter server-1b092822-0ccd-384f-79f0-bd4d6ddb280c 1b092822-0ccd-384f-79f0-bd4d6ddb280c}" testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.296Z [ERROR] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: failed to get log: index=23 error="log not found" testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.296Z [ERROR] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: failed to get log: index=23 error="log not found" testing.go:153: 2021-08-25T09:28:12.297Z [INFO] creating new snapshot: path=/tmp/raft104115317/snapshots/2-113-1629883692297.tmp testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.299Z [ERROR] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: failed to get log: index=23 error="log not found" testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.299Z [ERROR] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: failed to get log: index=23 error="log not found" testing.go:153: 2021-08-25T09:28:12.300Z [INFO] creating new snapshot: path=/tmp/raft134093935/snapshots/2-113-1629883692300.tmp testLoggerAdapter verbose: server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: 2021-08-25T09:28:12.324Z [INFO] server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: copied to local snapshot: bytes=71 testing.go:149: server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: 2021-08-25T09:28:12.324Z [INFO] server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: 2021-08-25T09:28:12.324Z [INFO] server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: Installed remote snapshot testing.go:149: server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: 2021-08-25T09:28:12.324Z [INFO] server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: Installed remote snapshot testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.325Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: pipelining replication: peer="{Voter server-1b092822-0ccd-384f-79f0-bd4d6ddb280c 1b092822-0ccd-384f-79f0-bd4d6ddb280c}" testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.325Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: pipelining replication: peer="{Voter server-1b092822-0ccd-384f-79f0-bd4d6ddb280c 1b092822-0ccd-384f-79f0-bd4d6ddb280c}" testLoggerAdapter verbose: server-fec26d77-9d21-8ed3-d991-338d6313021e: 2021-08-25T09:28:12.327Z [INFO] server-fec26d77-9d21-8ed3-d991-338d6313021e: copied to local snapshot: bytes=71 testing.go:149: server-fec26d77-9d21-8ed3-d991-338d6313021e: 2021-08-25T09:28:12.327Z [INFO] server-fec26d77-9d21-8ed3-d991-338d6313021e: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-fec26d77-9d21-8ed3-d991-338d6313021e: 2021-08-25T09:28:12.327Z [INFO] server-fec26d77-9d21-8ed3-d991-338d6313021e: Installed remote snapshot testing.go:149: server-fec26d77-9d21-8ed3-d991-338d6313021e: 2021-08-25T09:28:12.327Z [INFO] server-fec26d77-9d21-8ed3-d991-338d6313021e: Installed remote snapshot testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.328Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: pipelining replication: peer="{Voter server-fec26d77-9d21-8ed3-d991-338d6313021e fec26d77-9d21-8ed3-d991-338d6313021e}" testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.328Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: pipelining replication: peer="{Voter server-fec26d77-9d21-8ed3-d991-338d6313021e fec26d77-9d21-8ed3-d991-338d6313021e}" testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.341Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: aborting pipeline replication: peer="{Voter server-1b092822-0ccd-384f-79f0-bd4d6ddb280c 1b092822-0ccd-384f-79f0-bd4d6ddb280c}" testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.341Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: aborting pipeline replication: peer="{Voter server-1b092822-0ccd-384f-79f0-bd4d6ddb280c 1b092822-0ccd-384f-79f0-bd4d6ddb280c}" testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.341Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: aborting pipeline replication: peer="{Voter server-fec26d77-9d21-8ed3-d991-338d6313021e fec26d77-9d21-8ed3-d991-338d6313021e}" testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.341Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: aborting pipeline replication: peer="{Voter server-fec26d77-9d21-8ed3-d991-338d6313021e fec26d77-9d21-8ed3-d991-338d6313021e}" testLoggerAdapter verbose: server-867ef441-c97f-bde7-5cb1-206ad54eff9b: 2021-08-25T09:28:12.346Z [INFO] server-867ef441-c97f-bde7-5cb1-206ad54eff9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-867ef441-c97f-bde7-5cb1-206ad54eff9b Address:867ef441-c97f-bde7-5cb1-206ad54eff9b} {Suffrage:Voter ID:server-ffe8e498-96ff-07ce-8d4f-378576ebda6c Address:ffe8e498-96ff-07ce-8d4f-378576ebda6c} {Suffrage:Voter ID:server-a5f12987-1429-775d-a68f-b7499072dd13 Address:a5f12987-1429-775d-a68f-b7499072dd13}]" testing.go:149: server-867ef441-c97f-bde7-5cb1-206ad54eff9b: 2021-08-25T09:28:12.346Z [INFO] server-867ef441-c97f-bde7-5cb1-206ad54eff9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-867ef441-c97f-bde7-5cb1-206ad54eff9b Address:867ef441-c97f-bde7-5cb1-206ad54eff9b} {Suffrage:Voter ID:server-ffe8e498-96ff-07ce-8d4f-378576ebda6c Address:ffe8e498-96ff-07ce-8d4f-378576ebda6c} {Suffrage:Voter ID:server-a5f12987-1429-775d-a68f-b7499072dd13 Address:a5f12987-1429-775d-a68f-b7499072dd13}]" testLoggerAdapter verbose: server-867ef441-c97f-bde7-5cb1-206ad54eff9b: 2021-08-25T09:28:12.346Z [INFO] server-867ef441-c97f-bde7-5cb1-206ad54eff9b: entering follower state: follower="Node at 867ef441-c97f-bde7-5cb1-206ad54eff9b [Follower]" leader= testing.go:149: server-867ef441-c97f-bde7-5cb1-206ad54eff9b: 2021-08-25T09:28:12.346Z [INFO] server-867ef441-c97f-bde7-5cb1-206ad54eff9b: entering follower state: follower="Node at 867ef441-c97f-bde7-5cb1-206ad54eff9b [Follower]" leader= testLoggerAdapter verbose: server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: 2021-08-25T09:28:12.346Z [INFO] server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-867ef441-c97f-bde7-5cb1-206ad54eff9b Address:867ef441-c97f-bde7-5cb1-206ad54eff9b} {Suffrage:Voter ID:server-ffe8e498-96ff-07ce-8d4f-378576ebda6c Address:ffe8e498-96ff-07ce-8d4f-378576ebda6c} {Suffrage:Voter ID:server-a5f12987-1429-775d-a68f-b7499072dd13 Address:a5f12987-1429-775d-a68f-b7499072dd13}]" testing.go:149: server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: 2021-08-25T09:28:12.346Z [INFO] server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-867ef441-c97f-bde7-5cb1-206ad54eff9b Address:867ef441-c97f-bde7-5cb1-206ad54eff9b} {Suffrage:Voter ID:server-ffe8e498-96ff-07ce-8d4f-378576ebda6c Address:ffe8e498-96ff-07ce-8d4f-378576ebda6c} {Suffrage:Voter ID:server-a5f12987-1429-775d-a68f-b7499072dd13 Address:a5f12987-1429-775d-a68f-b7499072dd13}]" testLoggerAdapter verbose: server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: 2021-08-25T09:28:12.346Z [INFO] server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: entering follower state: follower="Node at ffe8e498-96ff-07ce-8d4f-378576ebda6c [Follower]" leader= testing.go:149: server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: 2021-08-25T09:28:12.346Z [INFO] server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: entering follower state: follower="Node at ffe8e498-96ff-07ce-8d4f-378576ebda6c [Follower]" leader= testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.346Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-867ef441-c97f-bde7-5cb1-206ad54eff9b Address:867ef441-c97f-bde7-5cb1-206ad54eff9b} {Suffrage:Voter ID:server-ffe8e498-96ff-07ce-8d4f-378576ebda6c Address:ffe8e498-96ff-07ce-8d4f-378576ebda6c} {Suffrage:Voter ID:server-a5f12987-1429-775d-a68f-b7499072dd13 Address:a5f12987-1429-775d-a68f-b7499072dd13}]" testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.346Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-867ef441-c97f-bde7-5cb1-206ad54eff9b Address:867ef441-c97f-bde7-5cb1-206ad54eff9b} {Suffrage:Voter ID:server-ffe8e498-96ff-07ce-8d4f-378576ebda6c Address:ffe8e498-96ff-07ce-8d4f-378576ebda6c} {Suffrage:Voter ID:server-a5f12987-1429-775d-a68f-b7499072dd13 Address:a5f12987-1429-775d-a68f-b7499072dd13}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:12.347Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:12.347Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.347Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: entering follower state: follower="Node at a5f12987-1429-775d-a68f-b7499072dd13 [Follower]" leader= testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.347Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: entering follower state: follower="Node at a5f12987-1429-775d-a68f-b7499072dd13 [Follower]" leader= testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.400Z [WARN] server-a5f12987-1429-775d-a68f-b7499072dd13: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.400Z [WARN] server-a5f12987-1429-775d-a68f-b7499072dd13: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.400Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: entering candidate state: node="Node at a5f12987-1429-775d-a68f-b7499072dd13 [Candidate]" term=2 testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.400Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: entering candidate state: node="Node at a5f12987-1429-775d-a68f-b7499072dd13 [Candidate]" term=2 testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.400Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: election won: tally=2 testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.400Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: election won: tally=2 testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: entering leader state: leader="Node at a5f12987-1429-775d-a68f-b7499072dd13 [Leader]" testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: entering leader state: leader="Node at a5f12987-1429-775d-a68f-b7499072dd13 [Leader]" testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: added peer, starting replication: peer=server-867ef441-c97f-bde7-5cb1-206ad54eff9b testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: added peer, starting replication: peer=server-867ef441-c97f-bde7-5cb1-206ad54eff9b testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: added peer, starting replication: peer=server-ffe8e498-96ff-07ce-8d4f-378576ebda6c testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: added peer, starting replication: peer=server-ffe8e498-96ff-07ce-8d4f-378576ebda6c testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: pipelining replication: peer="{Voter server-867ef441-c97f-bde7-5cb1-206ad54eff9b 867ef441-c97f-bde7-5cb1-206ad54eff9b}" testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: pipelining replication: peer="{Voter server-867ef441-c97f-bde7-5cb1-206ad54eff9b 867ef441-c97f-bde7-5cb1-206ad54eff9b}" testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: pipelining replication: peer="{Voter server-ffe8e498-96ff-07ce-8d4f-378576ebda6c ffe8e498-96ff-07ce-8d4f-378576ebda6c}" testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: pipelining replication: peer="{Voter server-ffe8e498-96ff-07ce-8d4f-378576ebda6c ffe8e498-96ff-07ce-8d4f-378576ebda6c}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:12.506Z [INFO] stable state for Leader reached at 2021-08-25 09:28:12.40121429 +0000 UTC m=+12.165430239 (1 nodes), 54.059755ms from start of poll, 55.270123ms from cluster start. Timeout at 2021-08-25 09:28:12.506354139 +0000 UTC m=+12.270570408, 105.140169ms after stability testing.go:149: cluster: 2021-08-25T09:28:12.506Z [INFO] stable state for Leader reached at 2021-08-25 09:28:12.40121429 +0000 UTC m=+12.165430239 (1 nodes), 54.059755ms from start of poll, 55.270123ms from cluster start. Timeout at 2021-08-25 09:28:12.506354139 +0000 UTC m=+12.270570408, 105.140169ms after stability testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.507Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: starting snapshot up to: index=12 testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.507Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: starting snapshot up to: index=12 testing.go:153: 2021-08-25T09:28:12.507Z [INFO] creating new snapshot: path=/tmp/raft493305847/snapshots/2-12-1629883692507.tmp testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.524Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: snapshot complete up to: index=12 testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.524Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: snapshot complete up to: index=12 testing.go:153: 2021-08-25T09:28:12.525Z [INFO] creating new snapshot: path=/tmp/raft493305847/snapshots/2-1013-1629883692525.tmp testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.542Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: copied to local snapshot: bytes=71 testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.542Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.542Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: restored user snapshot: index=1 testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.542Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: restored user snapshot: index=1 testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.543Z [ERROR] server-a5f12987-1429-775d-a68f-b7499072dd13: failed to get log: index=23 error="log not found" testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.543Z [ERROR] server-a5f12987-1429-775d-a68f-b7499072dd13: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.543Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: aborting pipeline replication: peer="{Voter server-867ef441-c97f-bde7-5cb1-206ad54eff9b 867ef441-c97f-bde7-5cb1-206ad54eff9b}" testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.543Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: aborting pipeline replication: peer="{Voter server-867ef441-c97f-bde7-5cb1-206ad54eff9b 867ef441-c97f-bde7-5cb1-206ad54eff9b}" testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.543Z [ERROR] server-a5f12987-1429-775d-a68f-b7499072dd13: failed to get log: index=23 error="log not found" testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.543Z [ERROR] server-a5f12987-1429-775d-a68f-b7499072dd13: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.543Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: aborting pipeline replication: peer="{Voter server-ffe8e498-96ff-07ce-8d4f-378576ebda6c ffe8e498-96ff-07ce-8d4f-378576ebda6c}" testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.543Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: aborting pipeline replication: peer="{Voter server-ffe8e498-96ff-07ce-8d4f-378576ebda6c ffe8e498-96ff-07ce-8d4f-378576ebda6c}" testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.548Z [ERROR] server-a5f12987-1429-775d-a68f-b7499072dd13: failed to get log: index=23 error="log not found" testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.548Z [ERROR] server-a5f12987-1429-775d-a68f-b7499072dd13: failed to get log: index=23 error="log not found" testing.go:153: 2021-08-25T09:28:12.549Z [INFO] creating new snapshot: path=/tmp/raft136100029/snapshots/2-1013-1629883692549.tmp testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.553Z [ERROR] server-a5f12987-1429-775d-a68f-b7499072dd13: failed to get log: index=23 error="log not found" testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.553Z [ERROR] server-a5f12987-1429-775d-a68f-b7499072dd13: failed to get log: index=23 error="log not found" testing.go:153: 2021-08-25T09:28:12.554Z [INFO] creating new snapshot: path=/tmp/raft573978899/snapshots/2-1013-1629883692554.tmp testLoggerAdapter verbose: server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: 2021-08-25T09:28:12.576Z [INFO] server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: copied to local snapshot: bytes=71 testing.go:149: server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: 2021-08-25T09:28:12.576Z [INFO] server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: 2021-08-25T09:28:12.577Z [INFO] server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: Installed remote snapshot testing.go:149: server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: 2021-08-25T09:28:12.577Z [INFO] server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: Installed remote snapshot testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.577Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: pipelining replication: peer="{Voter server-ffe8e498-96ff-07ce-8d4f-378576ebda6c ffe8e498-96ff-07ce-8d4f-378576ebda6c}" testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.577Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: pipelining replication: peer="{Voter server-ffe8e498-96ff-07ce-8d4f-378576ebda6c ffe8e498-96ff-07ce-8d4f-378576ebda6c}" testLoggerAdapter verbose: server-867ef441-c97f-bde7-5cb1-206ad54eff9b: 2021-08-25T09:28:12.581Z [INFO] server-867ef441-c97f-bde7-5cb1-206ad54eff9b: copied to local snapshot: bytes=71 testing.go:149: server-867ef441-c97f-bde7-5cb1-206ad54eff9b: 2021-08-25T09:28:12.581Z [INFO] server-867ef441-c97f-bde7-5cb1-206ad54eff9b: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-867ef441-c97f-bde7-5cb1-206ad54eff9b: 2021-08-25T09:28:12.581Z [INFO] server-867ef441-c97f-bde7-5cb1-206ad54eff9b: Installed remote snapshot testing.go:149: server-867ef441-c97f-bde7-5cb1-206ad54eff9b: 2021-08-25T09:28:12.581Z [INFO] server-867ef441-c97f-bde7-5cb1-206ad54eff9b: Installed remote snapshot testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.581Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: pipelining replication: peer="{Voter server-867ef441-c97f-bde7-5cb1-206ad54eff9b 867ef441-c97f-bde7-5cb1-206ad54eff9b}" testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.581Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: pipelining replication: peer="{Voter server-867ef441-c97f-bde7-5cb1-206ad54eff9b 867ef441-c97f-bde7-5cb1-206ad54eff9b}" testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.593Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: aborting pipeline replication: peer="{Voter server-ffe8e498-96ff-07ce-8d4f-378576ebda6c ffe8e498-96ff-07ce-8d4f-378576ebda6c}" testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.593Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: aborting pipeline replication: peer="{Voter server-ffe8e498-96ff-07ce-8d4f-378576ebda6c ffe8e498-96ff-07ce-8d4f-378576ebda6c}" testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.593Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: aborting pipeline replication: peer="{Voter server-867ef441-c97f-bde7-5cb1-206ad54eff9b 867ef441-c97f-bde7-5cb1-206ad54eff9b}" testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.593Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: aborting pipeline replication: peer="{Voter server-867ef441-c97f-bde7-5cb1-206ad54eff9b 867ef441-c97f-bde7-5cb1-206ad54eff9b}" testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.600Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29387dcd-909d-fc54-1781-d0ef41cb21f7 Address:29387dcd-909d-fc54-1781-d0ef41cb21f7} {Suffrage:Voter ID:server-e461fd51-53cd-29dc-4854-9240d5f8d499 Address:e461fd51-53cd-29dc-4854-9240d5f8d499} {Suffrage:Voter ID:server-794ef44c-b813-251f-1c50-a891b0f9e742 Address:794ef44c-b813-251f-1c50-a891b0f9e742}]" testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.600Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29387dcd-909d-fc54-1781-d0ef41cb21f7 Address:29387dcd-909d-fc54-1781-d0ef41cb21f7} {Suffrage:Voter ID:server-e461fd51-53cd-29dc-4854-9240d5f8d499 Address:e461fd51-53cd-29dc-4854-9240d5f8d499} {Suffrage:Voter ID:server-794ef44c-b813-251f-1c50-a891b0f9e742 Address:794ef44c-b813-251f-1c50-a891b0f9e742}]" testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.600Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: entering follower state: follower="Node at 29387dcd-909d-fc54-1781-d0ef41cb21f7 [Follower]" leader= testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.600Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: entering follower state: follower="Node at 29387dcd-909d-fc54-1781-d0ef41cb21f7 [Follower]" leader= testLoggerAdapter verbose: server-e461fd51-53cd-29dc-4854-9240d5f8d499: 2021-08-25T09:28:12.601Z [INFO] server-e461fd51-53cd-29dc-4854-9240d5f8d499: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29387dcd-909d-fc54-1781-d0ef41cb21f7 Address:29387dcd-909d-fc54-1781-d0ef41cb21f7} {Suffrage:Voter ID:server-e461fd51-53cd-29dc-4854-9240d5f8d499 Address:e461fd51-53cd-29dc-4854-9240d5f8d499} {Suffrage:Voter ID:server-794ef44c-b813-251f-1c50-a891b0f9e742 Address:794ef44c-b813-251f-1c50-a891b0f9e742}]" testing.go:149: server-e461fd51-53cd-29dc-4854-9240d5f8d499: 2021-08-25T09:28:12.601Z [INFO] server-e461fd51-53cd-29dc-4854-9240d5f8d499: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29387dcd-909d-fc54-1781-d0ef41cb21f7 Address:29387dcd-909d-fc54-1781-d0ef41cb21f7} {Suffrage:Voter ID:server-e461fd51-53cd-29dc-4854-9240d5f8d499 Address:e461fd51-53cd-29dc-4854-9240d5f8d499} {Suffrage:Voter ID:server-794ef44c-b813-251f-1c50-a891b0f9e742 Address:794ef44c-b813-251f-1c50-a891b0f9e742}]" testLoggerAdapter verbose: server-e461fd51-53cd-29dc-4854-9240d5f8d499: 2021-08-25T09:28:12.601Z [INFO] server-e461fd51-53cd-29dc-4854-9240d5f8d499: entering follower state: follower="Node at e461fd51-53cd-29dc-4854-9240d5f8d499 [Follower]" leader= testing.go:149: server-e461fd51-53cd-29dc-4854-9240d5f8d499: 2021-08-25T09:28:12.601Z [INFO] server-e461fd51-53cd-29dc-4854-9240d5f8d499: entering follower state: follower="Node at e461fd51-53cd-29dc-4854-9240d5f8d499 [Follower]" leader= testLoggerAdapter verbose: server-794ef44c-b813-251f-1c50-a891b0f9e742: 2021-08-25T09:28:12.601Z [INFO] server-794ef44c-b813-251f-1c50-a891b0f9e742: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29387dcd-909d-fc54-1781-d0ef41cb21f7 Address:29387dcd-909d-fc54-1781-d0ef41cb21f7} {Suffrage:Voter ID:server-e461fd51-53cd-29dc-4854-9240d5f8d499 Address:e461fd51-53cd-29dc-4854-9240d5f8d499} {Suffrage:Voter ID:server-794ef44c-b813-251f-1c50-a891b0f9e742 Address:794ef44c-b813-251f-1c50-a891b0f9e742}]" testing.go:149: server-794ef44c-b813-251f-1c50-a891b0f9e742: 2021-08-25T09:28:12.601Z [INFO] server-794ef44c-b813-251f-1c50-a891b0f9e742: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29387dcd-909d-fc54-1781-d0ef41cb21f7 Address:29387dcd-909d-fc54-1781-d0ef41cb21f7} {Suffrage:Voter ID:server-e461fd51-53cd-29dc-4854-9240d5f8d499 Address:e461fd51-53cd-29dc-4854-9240d5f8d499} {Suffrage:Voter ID:server-794ef44c-b813-251f-1c50-a891b0f9e742 Address:794ef44c-b813-251f-1c50-a891b0f9e742}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:12.601Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:12.601Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-794ef44c-b813-251f-1c50-a891b0f9e742: 2021-08-25T09:28:12.601Z [INFO] server-794ef44c-b813-251f-1c50-a891b0f9e742: entering follower state: follower="Node at 794ef44c-b813-251f-1c50-a891b0f9e742 [Follower]" leader= testing.go:149: server-794ef44c-b813-251f-1c50-a891b0f9e742: 2021-08-25T09:28:12.601Z [INFO] server-794ef44c-b813-251f-1c50-a891b0f9e742: entering follower state: follower="Node at 794ef44c-b813-251f-1c50-a891b0f9e742 [Follower]" leader= testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.662Z [WARN] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.662Z [WARN] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: entering candidate state: node="Node at 29387dcd-909d-fc54-1781-d0ef41cb21f7 [Candidate]" term=2 testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: entering candidate state: node="Node at 29387dcd-909d-fc54-1781-d0ef41cb21f7 [Candidate]" term=2 testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: election won: tally=2 testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: election won: tally=2 testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: entering leader state: leader="Node at 29387dcd-909d-fc54-1781-d0ef41cb21f7 [Leader]" testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: entering leader state: leader="Node at 29387dcd-909d-fc54-1781-d0ef41cb21f7 [Leader]" testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: added peer, starting replication: peer=server-e461fd51-53cd-29dc-4854-9240d5f8d499 testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: added peer, starting replication: peer=server-e461fd51-53cd-29dc-4854-9240d5f8d499 testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: added peer, starting replication: peer=server-794ef44c-b813-251f-1c50-a891b0f9e742 testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: added peer, starting replication: peer=server-794ef44c-b813-251f-1c50-a891b0f9e742 testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: pipelining replication: peer="{Voter server-e461fd51-53cd-29dc-4854-9240d5f8d499 e461fd51-53cd-29dc-4854-9240d5f8d499}" testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: pipelining replication: peer="{Voter server-e461fd51-53cd-29dc-4854-9240d5f8d499 e461fd51-53cd-29dc-4854-9240d5f8d499}" testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.664Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: pipelining replication: peer="{Voter server-794ef44c-b813-251f-1c50-a891b0f9e742 794ef44c-b813-251f-1c50-a891b0f9e742}" testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.664Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: pipelining replication: peer="{Voter server-794ef44c-b813-251f-1c50-a891b0f9e742 794ef44c-b813-251f-1c50-a891b0f9e742}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:12.768Z [INFO] stable state for Leader reached at 2021-08-25 09:28:12.663517232 +0000 UTC m=+12.427733161 (1 nodes), 61.800656ms from start of poll, 63.153891ms from cluster start. Timeout at 2021-08-25 09:28:12.768538691 +0000 UTC m=+12.532754660, 105.021499ms after stability testing.go:149: cluster: 2021-08-25T09:28:12.768Z [INFO] stable state for Leader reached at 2021-08-25 09:28:12.663517232 +0000 UTC m=+12.427733161 (1 nodes), 61.800656ms from start of poll, 63.153891ms from cluster start. Timeout at 2021-08-25 09:28:12.768538691 +0000 UTC m=+12.532754660, 105.021499ms after stability testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.769Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: starting snapshot up to: index=12 testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.769Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: starting snapshot up to: index=12 testing.go:153: 2021-08-25T09:28:12.769Z [INFO] creating new snapshot: path=/tmp/raft608534849/snapshots/2-12-1629883692769.tmp testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.788Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: snapshot complete up to: index=12 testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.788Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: snapshot complete up to: index=12 testing.go:153: 2021-08-25T09:28:12.789Z [INFO] creating new snapshot: path=/tmp/raft608534849/snapshots/2-10013-1629883692789.tmp testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.805Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: copied to local snapshot: bytes=71 testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.805Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: restored user snapshot: index=1 testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: restored user snapshot: index=1 testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [ERROR] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: failed to get log: index=23 error="log not found" testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [ERROR] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: aborting pipeline replication: peer="{Voter server-794ef44c-b813-251f-1c50-a891b0f9e742 794ef44c-b813-251f-1c50-a891b0f9e742}" testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: aborting pipeline replication: peer="{Voter server-794ef44c-b813-251f-1c50-a891b0f9e742 794ef44c-b813-251f-1c50-a891b0f9e742}" testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [ERROR] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: failed to get log: index=23 error="log not found" testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [ERROR] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: aborting pipeline replication: peer="{Voter server-e461fd51-53cd-29dc-4854-9240d5f8d499 e461fd51-53cd-29dc-4854-9240d5f8d499}" testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: aborting pipeline replication: peer="{Voter server-e461fd51-53cd-29dc-4854-9240d5f8d499 e461fd51-53cd-29dc-4854-9240d5f8d499}" testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.814Z [ERROR] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: failed to get log: index=23 error="log not found" testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.814Z [ERROR] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.815Z [ERROR] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: failed to get log: index=23 error="log not found" testing.go:153: 2021-08-25T09:28:12.815Z [INFO] creating new snapshot: path=/tmp/raft320300549/snapshots/2-10013-1629883692815.tmp testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.815Z [ERROR] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: failed to get log: index=23 error="log not found" testing.go:153: 2021-08-25T09:28:12.815Z [INFO] creating new snapshot: path=/tmp/raft356716315/snapshots/2-10013-1629883692815.tmp testLoggerAdapter verbose: server-794ef44c-b813-251f-1c50-a891b0f9e742: 2021-08-25T09:28:12.842Z [INFO] server-794ef44c-b813-251f-1c50-a891b0f9e742: copied to local snapshot: bytes=71 testing.go:149: server-794ef44c-b813-251f-1c50-a891b0f9e742: 2021-08-25T09:28:12.842Z [INFO] server-794ef44c-b813-251f-1c50-a891b0f9e742: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-794ef44c-b813-251f-1c50-a891b0f9e742: 2021-08-25T09:28:12.843Z [INFO] server-794ef44c-b813-251f-1c50-a891b0f9e742: Installed remote snapshot testing.go:149: server-794ef44c-b813-251f-1c50-a891b0f9e742: 2021-08-25T09:28:12.843Z [INFO] server-794ef44c-b813-251f-1c50-a891b0f9e742: Installed remote snapshot testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.843Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: pipelining replication: peer="{Voter server-794ef44c-b813-251f-1c50-a891b0f9e742 794ef44c-b813-251f-1c50-a891b0f9e742}" testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.843Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: pipelining replication: peer="{Voter server-794ef44c-b813-251f-1c50-a891b0f9e742 794ef44c-b813-251f-1c50-a891b0f9e742}" testLoggerAdapter verbose: server-e461fd51-53cd-29dc-4854-9240d5f8d499: 2021-08-25T09:28:12.845Z [INFO] server-e461fd51-53cd-29dc-4854-9240d5f8d499: copied to local snapshot: bytes=71 testing.go:149: server-e461fd51-53cd-29dc-4854-9240d5f8d499: 2021-08-25T09:28:12.845Z [INFO] server-e461fd51-53cd-29dc-4854-9240d5f8d499: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-e461fd51-53cd-29dc-4854-9240d5f8d499: 2021-08-25T09:28:12.846Z [INFO] server-e461fd51-53cd-29dc-4854-9240d5f8d499: Installed remote snapshot testing.go:149: server-e461fd51-53cd-29dc-4854-9240d5f8d499: 2021-08-25T09:28:12.846Z [INFO] server-e461fd51-53cd-29dc-4854-9240d5f8d499: Installed remote snapshot testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.846Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: pipelining replication: peer="{Voter server-e461fd51-53cd-29dc-4854-9240d5f8d499 e461fd51-53cd-29dc-4854-9240d5f8d499}" testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.846Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: pipelining replication: peer="{Voter server-e461fd51-53cd-29dc-4854-9240d5f8d499 e461fd51-53cd-29dc-4854-9240d5f8d499}" testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.860Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: aborting pipeline replication: peer="{Voter server-794ef44c-b813-251f-1c50-a891b0f9e742 794ef44c-b813-251f-1c50-a891b0f9e742}" testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.860Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: aborting pipeline replication: peer="{Voter server-794ef44c-b813-251f-1c50-a891b0f9e742 794ef44c-b813-251f-1c50-a891b0f9e742}" testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.860Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: aborting pipeline replication: peer="{Voter server-e461fd51-53cd-29dc-4854-9240d5f8d499 e461fd51-53cd-29dc-4854-9240d5f8d499}" testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.860Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: aborting pipeline replication: peer="{Voter server-e461fd51-53cd-29dc-4854-9240d5f8d499 e461fd51-53cd-29dc-4854-9240d5f8d499}" --- PASS: TestRaft_UserRestore (1.59s) === RUN TestRaft_SendSnapshotFollower testLoggerAdapter verbose: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:12.865Z [INFO] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a242f697-00d2-d8bf-2bcf-137be5f882fc Address:a242f697-00d2-d8bf-2bcf-137be5f882fc} {Suffrage:Voter ID:server-90ea17d4-ad59-3540-df78-af734963bf9f Address:90ea17d4-ad59-3540-df78-af734963bf9f} {Suffrage:Voter ID:server-1956d413-47c5-00aa-5727-c278aeaeb89b Address:1956d413-47c5-00aa-5727-c278aeaeb89b}]" testing.go:149: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:12.865Z [INFO] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a242f697-00d2-d8bf-2bcf-137be5f882fc Address:a242f697-00d2-d8bf-2bcf-137be5f882fc} {Suffrage:Voter ID:server-90ea17d4-ad59-3540-df78-af734963bf9f Address:90ea17d4-ad59-3540-df78-af734963bf9f} {Suffrage:Voter ID:server-1956d413-47c5-00aa-5727-c278aeaeb89b Address:1956d413-47c5-00aa-5727-c278aeaeb89b}]" testLoggerAdapter verbose: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:12.865Z [INFO] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: entering follower state: follower="Node at a242f697-00d2-d8bf-2bcf-137be5f882fc [Follower]" leader= testing.go:149: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:12.865Z [INFO] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: entering follower state: follower="Node at a242f697-00d2-d8bf-2bcf-137be5f882fc [Follower]" leader= testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.865Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a242f697-00d2-d8bf-2bcf-137be5f882fc Address:a242f697-00d2-d8bf-2bcf-137be5f882fc} {Suffrage:Voter ID:server-90ea17d4-ad59-3540-df78-af734963bf9f Address:90ea17d4-ad59-3540-df78-af734963bf9f} {Suffrage:Voter ID:server-1956d413-47c5-00aa-5727-c278aeaeb89b Address:1956d413-47c5-00aa-5727-c278aeaeb89b}]" testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.865Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a242f697-00d2-d8bf-2bcf-137be5f882fc Address:a242f697-00d2-d8bf-2bcf-137be5f882fc} {Suffrage:Voter ID:server-90ea17d4-ad59-3540-df78-af734963bf9f Address:90ea17d4-ad59-3540-df78-af734963bf9f} {Suffrage:Voter ID:server-1956d413-47c5-00aa-5727-c278aeaeb89b Address:1956d413-47c5-00aa-5727-c278aeaeb89b}]" testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.865Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: entering follower state: follower="Node at 90ea17d4-ad59-3540-df78-af734963bf9f [Follower]" leader= testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.865Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: entering follower state: follower="Node at 90ea17d4-ad59-3540-df78-af734963bf9f [Follower]" leader= testLoggerAdapter verbose: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:12.865Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a242f697-00d2-d8bf-2bcf-137be5f882fc Address:a242f697-00d2-d8bf-2bcf-137be5f882fc} {Suffrage:Voter ID:server-90ea17d4-ad59-3540-df78-af734963bf9f Address:90ea17d4-ad59-3540-df78-af734963bf9f} {Suffrage:Voter ID:server-1956d413-47c5-00aa-5727-c278aeaeb89b Address:1956d413-47c5-00aa-5727-c278aeaeb89b}]" testing.go:149: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:12.865Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a242f697-00d2-d8bf-2bcf-137be5f882fc Address:a242f697-00d2-d8bf-2bcf-137be5f882fc} {Suffrage:Voter ID:server-90ea17d4-ad59-3540-df78-af734963bf9f Address:90ea17d4-ad59-3540-df78-af734963bf9f} {Suffrage:Voter ID:server-1956d413-47c5-00aa-5727-c278aeaeb89b Address:1956d413-47c5-00aa-5727-c278aeaeb89b}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:12.866Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:12.866Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:12.866Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: entering follower state: follower="Node at 1956d413-47c5-00aa-5727-c278aeaeb89b [Follower]" leader= testing.go:149: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:12.866Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: entering follower state: follower="Node at 1956d413-47c5-00aa-5727-c278aeaeb89b [Follower]" leader= testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [WARN] server-90ea17d4-ad59-3540-df78-af734963bf9f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [WARN] server-90ea17d4-ad59-3540-df78-af734963bf9f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: entering candidate state: node="Node at 90ea17d4-ad59-3540-df78-af734963bf9f [Candidate]" term=2 testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: entering candidate state: node="Node at 90ea17d4-ad59-3540-df78-af734963bf9f [Candidate]" term=2 testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: election won: tally=2 testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: election won: tally=2 testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: entering leader state: leader="Node at 90ea17d4-ad59-3540-df78-af734963bf9f [Leader]" testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: entering leader state: leader="Node at 90ea17d4-ad59-3540-df78-af734963bf9f [Leader]" testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: added peer, starting replication: peer=server-a242f697-00d2-d8bf-2bcf-137be5f882fc testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: added peer, starting replication: peer=server-a242f697-00d2-d8bf-2bcf-137be5f882fc testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: added peer, starting replication: peer=server-1956d413-47c5-00aa-5727-c278aeaeb89b testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: added peer, starting replication: peer=server-1956d413-47c5-00aa-5727-c278aeaeb89b testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.927Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: pipelining replication: peer="{Voter server-1956d413-47c5-00aa-5727-c278aeaeb89b 1956d413-47c5-00aa-5727-c278aeaeb89b}" testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.927Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: pipelining replication: peer="{Voter server-1956d413-47c5-00aa-5727-c278aeaeb89b 1956d413-47c5-00aa-5727-c278aeaeb89b}" testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.927Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: pipelining replication: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.927Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: pipelining replication: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.032Z [INFO] stable state for Follower reached at 2021-08-25 09:28:12.926919251 +0000 UTC m=+12.691135160 (2 nodes), 60.803149ms from start of poll, 62.069531ms from cluster start. Timeout at 2021-08-25 09:28:13.032024243 +0000 UTC m=+12.796240172, 105.105012ms after stability testing.go:149: cluster: 2021-08-25T09:28:13.032Z [INFO] stable state for Follower reached at 2021-08-25 09:28:12.926919251 +0000 UTC m=+12.691135160 (2 nodes), 60.803149ms from start of poll, 62.069531ms from cluster start. Timeout at 2021-08-25 09:28:13.032024243 +0000 UTC m=+12.796240172, 105.105012ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.032Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:13.032Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.137Z [INFO] stable state for Leader reached at 2021-08-25 09:28:13.032299937 +0000 UTC m=+12.796515826 (1 nodes), 1.979µs from start of poll, 167.450197ms from cluster start. Timeout at 2021-08-25 09:28:13.137444865 +0000 UTC m=+12.901660834, 105.145008ms after stability testing.go:149: cluster: 2021-08-25T09:28:13.137Z [INFO] stable state for Leader reached at 2021-08-25 09:28:13.032299937 +0000 UTC m=+12.796515826 (1 nodes), 1.979µs from start of poll, 167.450197ms from cluster start. Timeout at 2021-08-25 09:28:13.137444865 +0000 UTC m=+12.901660834, 105.145008ms after stability testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.137Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to pipeline appendEntries: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" error="append pipeline closed" testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.137Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to pipeline appendEntries: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" error="append pipeline closed" testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.138Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to heartbeat to: peer=a242f697-00d2-d8bf-2bcf-137be5f882fc error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.138Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to heartbeat to: peer=a242f697-00d2-d8bf-2bcf-137be5f882fc error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.138Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: aborting pipeline replication: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.138Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: aborting pipeline replication: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.138Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to appendEntries to: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.138Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to appendEntries to: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" raft_test.go:1320: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:13.141Z [ERROR] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:13.141Z [ERROR] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.142Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: starting snapshot up to: index=102 testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.142Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: starting snapshot up to: index=102 testing.go:153: 2021-08-25T09:28:13.142Z [INFO] creating new snapshot: path=/tmp/raft868723465/snapshots/2-102-1629883693142.tmp testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.149Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to appendEntries to: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.149Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to appendEntries to: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.154Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to heartbeat to: peer=a242f697-00d2-d8bf-2bcf-137be5f882fc error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.154Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to heartbeat to: peer=a242f697-00d2-d8bf-2bcf-137be5f882fc error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.160Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to appendEntries to: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.160Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to appendEntries to: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.160Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: compacting logs: from=1 to=92 testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.160Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.161Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: snapshot complete up to: index=102 testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.161Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: snapshot complete up to: index=102 testLoggerAdapter verbose: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:13.161Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: starting snapshot up to: index=102 testing.go:149: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:13.161Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: starting snapshot up to: index=102 testing.go:153: 2021-08-25T09:28:13.161Z [INFO] creating new snapshot: path=/tmp/raft991231523/snapshots/2-102-1629883693161.tmp testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.173Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to heartbeat to: peer=a242f697-00d2-d8bf-2bcf-137be5f882fc error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.173Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to heartbeat to: peer=a242f697-00d2-d8bf-2bcf-137be5f882fc error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" testLoggerAdapter verbose: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:13.177Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: compacting logs: from=1 to=92 testing.go:149: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:13.177Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:13.178Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: snapshot complete up to: index=102 testing.go:149: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:13.178Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: snapshot complete up to: index=102 testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.181Z [WARN] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to contact: server-id=server-a242f697-00d2-d8bf-2bcf-137be5f882fc time=50.180976ms testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.181Z [WARN] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to contact: server-id=server-a242f697-00d2-d8bf-2bcf-137be5f882fc time=50.180976ms testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.190Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to get log: index=2 error="log not found" testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.190Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to get log: index=2 error="log not found" testing.go:153: 2021-08-25T09:28:13.191Z [INFO] creating new snapshot: path=/tmp/raft944482431/snapshots/2-102-1629883693191.tmp testLoggerAdapter verbose: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:13.210Z [INFO] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: copied to local snapshot: bytes=693 testing.go:149: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:13.210Z [INFO] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:13.211Z [INFO] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: Installed remote snapshot testing.go:149: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:13.211Z [INFO] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: Installed remote snapshot testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.215Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: aborting pipeline replication: peer="{Voter server-1956d413-47c5-00aa-5727-c278aeaeb89b 1956d413-47c5-00aa-5727-c278aeaeb89b}" testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.215Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: aborting pipeline replication: peer="{Voter server-1956d413-47c5-00aa-5727-c278aeaeb89b 1956d413-47c5-00aa-5727-c278aeaeb89b}" --- PASS: TestRaft_SendSnapshotFollower (0.35s) === RUN TestRaft_SendSnapshotAndLogsFollower testLoggerAdapter verbose: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.219Z [INFO] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6fdd995-74de-79cc-3395-f2acc9b7508a Address:d6fdd995-74de-79cc-3395-f2acc9b7508a} {Suffrage:Voter ID:server-d879deee-1498-c8d0-09f8-225d22a28293 Address:d879deee-1498-c8d0-09f8-225d22a28293} {Suffrage:Voter ID:server-371e187f-c0a7-77bf-207e-183b35ce8bc9 Address:371e187f-c0a7-77bf-207e-183b35ce8bc9}]" testing.go:149: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.219Z [INFO] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6fdd995-74de-79cc-3395-f2acc9b7508a Address:d6fdd995-74de-79cc-3395-f2acc9b7508a} {Suffrage:Voter ID:server-d879deee-1498-c8d0-09f8-225d22a28293 Address:d879deee-1498-c8d0-09f8-225d22a28293} {Suffrage:Voter ID:server-371e187f-c0a7-77bf-207e-183b35ce8bc9 Address:371e187f-c0a7-77bf-207e-183b35ce8bc9}]" testLoggerAdapter verbose: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.219Z [INFO] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: entering follower state: follower="Node at d6fdd995-74de-79cc-3395-f2acc9b7508a [Follower]" leader= testing.go:149: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.219Z [INFO] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: entering follower state: follower="Node at d6fdd995-74de-79cc-3395-f2acc9b7508a [Follower]" leader= testLoggerAdapter verbose: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.220Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6fdd995-74de-79cc-3395-f2acc9b7508a Address:d6fdd995-74de-79cc-3395-f2acc9b7508a} {Suffrage:Voter ID:server-d879deee-1498-c8d0-09f8-225d22a28293 Address:d879deee-1498-c8d0-09f8-225d22a28293} {Suffrage:Voter ID:server-371e187f-c0a7-77bf-207e-183b35ce8bc9 Address:371e187f-c0a7-77bf-207e-183b35ce8bc9}]" testing.go:149: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.220Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6fdd995-74de-79cc-3395-f2acc9b7508a Address:d6fdd995-74de-79cc-3395-f2acc9b7508a} {Suffrage:Voter ID:server-d879deee-1498-c8d0-09f8-225d22a28293 Address:d879deee-1498-c8d0-09f8-225d22a28293} {Suffrage:Voter ID:server-371e187f-c0a7-77bf-207e-183b35ce8bc9 Address:371e187f-c0a7-77bf-207e-183b35ce8bc9}]" testLoggerAdapter verbose: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.220Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: entering follower state: follower="Node at d879deee-1498-c8d0-09f8-225d22a28293 [Follower]" leader= testing.go:149: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.220Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: entering follower state: follower="Node at d879deee-1498-c8d0-09f8-225d22a28293 [Follower]" leader= testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.220Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6fdd995-74de-79cc-3395-f2acc9b7508a Address:d6fdd995-74de-79cc-3395-f2acc9b7508a} {Suffrage:Voter ID:server-d879deee-1498-c8d0-09f8-225d22a28293 Address:d879deee-1498-c8d0-09f8-225d22a28293} {Suffrage:Voter ID:server-371e187f-c0a7-77bf-207e-183b35ce8bc9 Address:371e187f-c0a7-77bf-207e-183b35ce8bc9}]" testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.220Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6fdd995-74de-79cc-3395-f2acc9b7508a Address:d6fdd995-74de-79cc-3395-f2acc9b7508a} {Suffrage:Voter ID:server-d879deee-1498-c8d0-09f8-225d22a28293 Address:d879deee-1498-c8d0-09f8-225d22a28293} {Suffrage:Voter ID:server-371e187f-c0a7-77bf-207e-183b35ce8bc9 Address:371e187f-c0a7-77bf-207e-183b35ce8bc9}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.220Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.220Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: entering follower state: follower="Node at 371e187f-c0a7-77bf-207e-183b35ce8bc9 [Follower]" leader= testing.go:149: cluster: 2021-08-25T09:28:13.220Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.220Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: entering follower state: follower="Node at 371e187f-c0a7-77bf-207e-183b35ce8bc9 [Follower]" leader= testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.272Z [WARN] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.272Z [WARN] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: entering candidate state: node="Node at 371e187f-c0a7-77bf-207e-183b35ce8bc9 [Candidate]" term=2 testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: entering candidate state: node="Node at 371e187f-c0a7-77bf-207e-183b35ce8bc9 [Candidate]" term=2 testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: election won: tally=2 testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: election won: tally=2 testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: entering leader state: leader="Node at 371e187f-c0a7-77bf-207e-183b35ce8bc9 [Leader]" testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: entering leader state: leader="Node at 371e187f-c0a7-77bf-207e-183b35ce8bc9 [Leader]" testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: added peer, starting replication: peer=server-d6fdd995-74de-79cc-3395-f2acc9b7508a testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: added peer, starting replication: peer=server-d6fdd995-74de-79cc-3395-f2acc9b7508a testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: added peer, starting replication: peer=server-d879deee-1498-c8d0-09f8-225d22a28293 testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: added peer, starting replication: peer=server-d879deee-1498-c8d0-09f8-225d22a28293 testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: pipelining replication: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: pipelining replication: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: pipelining replication: peer="{Voter server-d879deee-1498-c8d0-09f8-225d22a28293 d879deee-1498-c8d0-09f8-225d22a28293}" testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: pipelining replication: peer="{Voter server-d879deee-1498-c8d0-09f8-225d22a28293 d879deee-1498-c8d0-09f8-225d22a28293}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.378Z [INFO] stable state for Follower reached at 2021-08-25 09:28:13.273443206 +0000 UTC m=+13.037659135 (2 nodes), 52.775875ms from start of poll, 53.980263ms from cluster start. Timeout at 2021-08-25 09:28:13.37852422 +0000 UTC m=+13.142740188, 105.081053ms after stability testing.go:149: cluster: 2021-08-25T09:28:13.378Z [INFO] stable state for Follower reached at 2021-08-25 09:28:13.273443206 +0000 UTC m=+13.037659135 (2 nodes), 52.775875ms from start of poll, 53.980263ms from cluster start. Timeout at 2021-08-25 09:28:13.37852422 +0000 UTC m=+13.142740188, 105.081053ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.378Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:13.378Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.484Z [INFO] stable state for Leader reached at 2021-08-25 09:28:13.378781656 +0000 UTC m=+13.142997525 (1 nodes), 1.8µs from start of poll, 159.318653ms from cluster start. Timeout at 2021-08-25 09:28:13.483931323 +0000 UTC m=+13.248147272, 105.149747ms after stability testing.go:149: cluster: 2021-08-25T09:28:13.484Z [INFO] stable state for Leader reached at 2021-08-25 09:28:13.378781656 +0000 UTC m=+13.142997525 (1 nodes), 1.8µs from start of poll, 159.318653ms from cluster start. Timeout at 2021-08-25 09:28:13.483931323 +0000 UTC m=+13.248147272, 105.149747ms after stability testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.484Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to pipeline appendEntries: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" error="append pipeline closed" testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.484Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to pipeline appendEntries: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" error="append pipeline closed" testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.484Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: aborting pipeline replication: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.484Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: aborting pipeline replication: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.484Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to appendEntries to: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.484Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to appendEntries to: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" raft_test.go:1362: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.487Z [ERROR] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.487Z [ERROR] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.487Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: starting snapshot up to: index=101 testing.go:149: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.487Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: starting snapshot up to: index=101 testing.go:153: 2021-08-25T09:28:13.487Z [INFO] creating new snapshot: path=/tmp/raft663820295/snapshots/2-101-1629883693487.tmp testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.488Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to heartbeat to: peer=d6fdd995-74de-79cc-3395-f2acc9b7508a error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.488Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to heartbeat to: peer=d6fdd995-74de-79cc-3395-f2acc9b7508a error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.494Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to appendEntries to: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.494Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to appendEntries to: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.505Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to appendEntries to: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.505Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to appendEntries to: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" testLoggerAdapter verbose: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.506Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: compacting logs: from=1 to=92 testing.go:149: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.506Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.506Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: snapshot complete up to: index=101 testing.go:149: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.506Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: snapshot complete up to: index=101 testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.506Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: starting snapshot up to: index=102 testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.506Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: starting snapshot up to: index=102 testing.go:153: 2021-08-25T09:28:13.506Z [INFO] creating new snapshot: path=/tmp/raft833142737/snapshots/2-102-1629883693506.tmp testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.506Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to heartbeat to: peer=d6fdd995-74de-79cc-3395-f2acc9b7508a error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.506Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to heartbeat to: peer=d6fdd995-74de-79cc-3395-f2acc9b7508a error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.524Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: compacting logs: from=1 to=92 testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.524Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.524Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: snapshot complete up to: index=102 testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.524Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: snapshot complete up to: index=102 testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.528Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to heartbeat to: peer=d6fdd995-74de-79cc-3395-f2acc9b7508a error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.528Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to heartbeat to: peer=d6fdd995-74de-79cc-3395-f2acc9b7508a error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" raft_test.go:1383: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.532Z [WARN] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to contact: server-id=server-d6fdd995-74de-79cc-3395-f2acc9b7508a time=50.192335ms testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.532Z [WARN] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to contact: server-id=server-d6fdd995-74de-79cc-3395-f2acc9b7508a time=50.192335ms testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.532Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to get log: index=2 error="log not found" testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.532Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to get log: index=2 error="log not found" testing.go:153: 2021-08-25T09:28:13.533Z [INFO] creating new snapshot: path=/tmp/raft226398285/snapshots/2-102-1629883693533.tmp testLoggerAdapter verbose: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.550Z [INFO] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: copied to local snapshot: bytes=693 testing.go:149: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.550Z [INFO] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.551Z [INFO] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: Installed remote snapshot testing.go:149: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.551Z [INFO] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: Installed remote snapshot testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.551Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: pipelining replication: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.551Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: pipelining replication: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.556Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: aborting pipeline replication: peer="{Voter server-d879deee-1498-c8d0-09f8-225d22a28293 d879deee-1498-c8d0-09f8-225d22a28293}" testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.556Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: aborting pipeline replication: peer="{Voter server-d879deee-1498-c8d0-09f8-225d22a28293 d879deee-1498-c8d0-09f8-225d22a28293}" testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.556Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: aborting pipeline replication: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.556Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: aborting pipeline replication: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.606Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to heartbeat to: peer=d6fdd995-74de-79cc-3395-f2acc9b7508a error="command timed out" testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.606Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to heartbeat to: peer=d6fdd995-74de-79cc-3395-f2acc9b7508a error="command timed out" --- PASS: TestRaft_SendSnapshotAndLogsFollower (0.39s) === RUN TestRaft_ReJoinFollower testLoggerAdapter verbose: server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: 2021-08-25T09:28:13.610Z [INFO] server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 Address:ef4138ce-d4e7-30c7-623a-85f9f00f8740} {Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8}]" testing.go:149: server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: 2021-08-25T09:28:13.610Z [INFO] server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 Address:ef4138ce-d4e7-30c7-623a-85f9f00f8740} {Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8}]" testLoggerAdapter verbose: server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: 2021-08-25T09:28:13.611Z [INFO] server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: entering follower state: follower="Node at ef4138ce-d4e7-30c7-623a-85f9f00f8740 [Follower]" leader= testing.go:149: server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: 2021-08-25T09:28:13.611Z [INFO] server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: entering follower state: follower="Node at ef4138ce-d4e7-30c7-623a-85f9f00f8740 [Follower]" leader= testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.611Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 Address:ef4138ce-d4e7-30c7-623a-85f9f00f8740} {Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8}]" testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.611Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 Address:ef4138ce-d4e7-30c7-623a-85f9f00f8740} {Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8}]" testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.611Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: entering follower state: follower="Node at 08a48afc-4e7f-e0ea-889d-60485e66883c [Follower]" leader= testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.611Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: entering follower state: follower="Node at 08a48afc-4e7f-e0ea-889d-60485e66883c [Follower]" leader= testLoggerAdapter verbose: server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8: 2021-08-25T09:28:13.611Z [INFO] server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 Address:ef4138ce-d4e7-30c7-623a-85f9f00f8740} {Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8}]" testing.go:149: server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8: 2021-08-25T09:28:13.611Z [INFO] server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 Address:ef4138ce-d4e7-30c7-623a-85f9f00f8740} {Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.611Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:13.611Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8: 2021-08-25T09:28:13.611Z [INFO] server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8: entering follower state: follower="Node at fd0ad873-2e1b-a020-190e-ee6147ed4eb8 [Follower]" leader= testing.go:149: server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8: 2021-08-25T09:28:13.611Z [INFO] server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8: entering follower state: follower="Node at fd0ad873-2e1b-a020-190e-ee6147ed4eb8 [Follower]" leader= testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.691Z [WARN] server-08a48afc-4e7f-e0ea-889d-60485e66883c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.691Z [WARN] server-08a48afc-4e7f-e0ea-889d-60485e66883c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.691Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: entering candidate state: node="Node at 08a48afc-4e7f-e0ea-889d-60485e66883c [Candidate]" term=2 testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.691Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: entering candidate state: node="Node at 08a48afc-4e7f-e0ea-889d-60485e66883c [Candidate]" term=2 testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: election won: tally=2 testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: election won: tally=2 testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: entering leader state: leader="Node at 08a48afc-4e7f-e0ea-889d-60485e66883c [Leader]" testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: entering leader state: leader="Node at 08a48afc-4e7f-e0ea-889d-60485e66883c [Leader]" testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: added peer, starting replication: peer=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: added peer, starting replication: peer=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: added peer, starting replication: peer=server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: added peer, starting replication: peer=server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: pipelining replication: peer="{Voter server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 fd0ad873-2e1b-a020-190e-ee6147ed4eb8}" testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: pipelining replication: peer="{Voter server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 fd0ad873-2e1b-a020-190e-ee6147ed4eb8}" testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: pipelining replication: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: pipelining replication: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.798Z [INFO] stable state for Leader reached at 2021-08-25 09:28:13.69281025 +0000 UTC m=+13.457026139 (1 nodes), 81.02523ms from start of poll, 82.085131ms from cluster start. Timeout at 2021-08-25 09:28:13.7979282 +0000 UTC m=+13.562144149, 105.11801ms after stability testing.go:149: cluster: 2021-08-25T09:28:13.798Z [INFO] stable state for Leader reached at 2021-08-25 09:28:13.69281025 +0000 UTC m=+13.457026139 (1 nodes), 81.02523ms from start of poll, 82.085131ms from cluster start. Timeout at 2021-08-25 09:28:13.7979282 +0000 UTC m=+13.562144149, 105.11801ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.803Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:13.803Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.908Z [INFO] stable state for Follower reached at 2021-08-25 09:28:13.803380214 +0000 UTC m=+13.567596123 (2 nodes), 3.32µs from start of poll, 192.655115ms from cluster start. Timeout at 2021-08-25 09:28:13.908528881 +0000 UTC m=+13.672744850, 105.148727ms after stability testing.go:149: cluster: 2021-08-25T09:28:13.908Z [INFO] stable state for Follower reached at 2021-08-25 09:28:13.803380214 +0000 UTC m=+13.567596123 (2 nodes), 3.32µs from start of poll, 192.655115ms from cluster start. Timeout at 2021-08-25 09:28:13.908528881 +0000 UTC m=+13.672744850, 105.148727ms after stability testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.908Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: updating configuration: command=RemoveServer server-id=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 server-addr= servers="[{Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8}]" testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.908Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: updating configuration: command=RemoveServer server-id=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 server-addr= servers="[{Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8}]" testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.909Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: removed peer, stopping replication: peer=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 last-index=3 testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.909Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: removed peer, stopping replication: peer=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 last-index=3 testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.909Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: aborting pipeline replication: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.909Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: aborting pipeline replication: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.019Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: updating configuration: command=AddStaging server-id=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 server-addr=ef4138ce-d4e7-30c7-623a-85f9f00f8740 servers="[{Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8} {Suffrage:Voter ID:server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 Address:ef4138ce-d4e7-30c7-623a-85f9f00f8740}]" testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.019Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: updating configuration: command=AddStaging server-id=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 server-addr=ef4138ce-d4e7-30c7-623a-85f9f00f8740 servers="[{Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8} {Suffrage:Voter ID:server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 Address:ef4138ce-d4e7-30c7-623a-85f9f00f8740}]" testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.020Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: added peer, starting replication: peer=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.020Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: added peer, starting replication: peer=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 testLoggerAdapter verbose: server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: 2021-08-25T09:28:14.020Z [WARN] server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: failed to get previous log: previous-index=4 last-index=3 error="log not found" testing.go:149: server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: 2021-08-25T09:28:14.020Z [WARN] server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: failed to get previous log: previous-index=4 last-index=3 error="log not found" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.020Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:14.020Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.020Z [WARN] server-08a48afc-4e7f-e0ea-889d-60485e66883c: appendEntries rejected, sending older logs: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" next=4 testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.020Z [WARN] server-08a48afc-4e7f-e0ea-889d-60485e66883c: appendEntries rejected, sending older logs: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" next=4 testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.020Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: pipelining replication: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.020Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: pipelining replication: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.125Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.020561449 +0000 UTC m=+13.784777338 (1 nodes), 1.52µs from start of poll, 409.83633ms from cluster start. Timeout at 2021-08-25 09:28:14.125733834 +0000 UTC m=+13.889949763, 105.172425ms after stability testing.go:149: cluster: 2021-08-25T09:28:14.125Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.020561449 +0000 UTC m=+13.784777338 (1 nodes), 1.52µs from start of poll, 409.83633ms from cluster start. Timeout at 2021-08-25 09:28:14.125733834 +0000 UTC m=+13.889949763, 105.172425ms after stability testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.126Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: aborting pipeline replication: peer="{Voter server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 fd0ad873-2e1b-a020-190e-ee6147ed4eb8}" testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.126Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: aborting pipeline replication: peer="{Voter server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 fd0ad873-2e1b-a020-190e-ee6147ed4eb8}" testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.126Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: aborting pipeline replication: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.126Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: aborting pipeline replication: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" --- PASS: TestRaft_ReJoinFollower (0.52s) === RUN TestRaft_LeaderLeaseExpire testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.130Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-056c5c14-3305-18be-9824-f917cef0ab0e Address:056c5c14-3305-18be-9824-f917cef0ab0e} {Suffrage:Voter ID:server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 Address:0f74db2c-ded7-6807-fe92-1bb9c00da2f8}]" testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.130Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-056c5c14-3305-18be-9824-f917cef0ab0e Address:056c5c14-3305-18be-9824-f917cef0ab0e} {Suffrage:Voter ID:server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 Address:0f74db2c-ded7-6807-fe92-1bb9c00da2f8}]" testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.130Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: entering follower state: follower="Node at 056c5c14-3305-18be-9824-f917cef0ab0e [Follower]" leader= testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.130Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: entering follower state: follower="Node at 056c5c14-3305-18be-9824-f917cef0ab0e [Follower]" leader= testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.130Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-056c5c14-3305-18be-9824-f917cef0ab0e Address:056c5c14-3305-18be-9824-f917cef0ab0e} {Suffrage:Voter ID:server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 Address:0f74db2c-ded7-6807-fe92-1bb9c00da2f8}]" testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.130Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-056c5c14-3305-18be-9824-f917cef0ab0e Address:056c5c14-3305-18be-9824-f917cef0ab0e} {Suffrage:Voter ID:server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 Address:0f74db2c-ded7-6807-fe92-1bb9c00da2f8}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.130Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:14.130Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.130Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering follower state: follower="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Follower]" leader= testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.130Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering follower state: follower="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Follower]" leader= testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.220Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.220Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.220Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering candidate state: node="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Candidate]" term=2 testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.220Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering candidate state: node="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Candidate]" term=2 testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.221Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: election won: tally=2 testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.221Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: election won: tally=2 testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.221Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering leader state: leader="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Leader]" testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.221Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering leader state: leader="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Leader]" testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.221Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: added peer, starting replication: peer=server-056c5c14-3305-18be-9824-f917cef0ab0e testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.221Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: added peer, starting replication: peer=server-056c5c14-3305-18be-9824-f917cef0ab0e testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.221Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: pipelining replication: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.221Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: pipelining replication: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.326Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.221477996 +0000 UTC m=+13.985693925 (1 nodes), 90.740347ms from start of poll, 91.704797ms from cluster start. Timeout at 2021-08-25 09:28:14.326609765 +0000 UTC m=+14.090825834, 105.131909ms after stability testing.go:149: cluster: 2021-08-25T09:28:14.326Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.221477996 +0000 UTC m=+13.985693925 (1 nodes), 90.740347ms from start of poll, 91.704797ms from cluster start. Timeout at 2021-08-25 09:28:14.326609765 +0000 UTC m=+14.090825834, 105.131909ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.332Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:14.332Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.437Z [INFO] stable state for Follower reached at 2021-08-25 09:28:14.332183087 +0000 UTC m=+14.096398996 (1 nodes), 2.58µs from start of poll, 202.409868ms from cluster start. Timeout at 2021-08-25 09:28:14.437336614 +0000 UTC m=+14.201552543, 105.153547ms after stability testing.go:149: cluster: 2021-08-25T09:28:14.437Z [INFO] stable state for Follower reached at 2021-08-25 09:28:14.332183087 +0000 UTC m=+14.096398996 (1 nodes), 2.58µs from start of poll, 202.409868ms from cluster start. Timeout at 2021-08-25 09:28:14.437336614 +0000 UTC m=+14.201552543, 105.153547ms after stability raft_test.go:1492: [INFO] Disconnecting Node at 056c5c14-3305-18be-9824-f917cef0ab0e [Follower] testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.441Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to heartbeat to: peer=056c5c14-3305-18be-9824-f917cef0ab0e error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.441Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to heartbeat to: peer=056c5c14-3305-18be-9824-f917cef0ab0e error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.441Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to pipeline appendEntries: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="append pipeline closed" testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.441Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to pipeline appendEntries: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="append pipeline closed" testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.442Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: aborting pipeline replication: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.442Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: aborting pipeline replication: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.450Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to appendEntries to: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.450Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to appendEntries to: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.457Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to heartbeat to: peer=056c5c14-3305-18be-9824-f917cef0ab0e error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.457Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to heartbeat to: peer=056c5c14-3305-18be-9824-f917cef0ab0e error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.467Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to appendEntries to: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.467Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to appendEntries to: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.475Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to heartbeat to: peer=056c5c14-3305-18be-9824-f917cef0ab0e error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.475Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to heartbeat to: peer=056c5c14-3305-18be-9824-f917cef0ab0e error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.482Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to appendEntries to: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.482Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to appendEntries to: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.486Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to contact: server-id=server-056c5c14-3305-18be-9824-f917cef0ab0e time=50.164197ms testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.486Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to contact: server-id=server-056c5c14-3305-18be-9824-f917cef0ab0e time=50.164197ms testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.486Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to contact quorum of nodes, stepping down testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.486Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.487Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering follower state: follower="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Follower]" leader= testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.487Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering follower state: follower="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Follower]" leader= testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.487Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:14.487Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.497Z [WARN] server-056c5c14-3305-18be-9824-f917cef0ab0e: heartbeat timeout reached, starting election: last-leader=0f74db2c-ded7-6807-fe92-1bb9c00da2f8 testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.497Z [WARN] server-056c5c14-3305-18be-9824-f917cef0ab0e: heartbeat timeout reached, starting election: last-leader=0f74db2c-ded7-6807-fe92-1bb9c00da2f8 testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.497Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: entering candidate state: node="Node at 056c5c14-3305-18be-9824-f917cef0ab0e [Candidate]" term=3 testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.497Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: entering candidate state: node="Node at 056c5c14-3305-18be-9824-f917cef0ab0e [Candidate]" term=3 testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.497Z [ERROR] server-056c5c14-3305-18be-9824-f917cef0ab0e: failed to make requestVote RPC: target="{Voter server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 0f74db2c-ded7-6807-fe92-1bb9c00da2f8}" error="failed to connect to peer: 0f74db2c-ded7-6807-fe92-1bb9c00da2f8" testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.497Z [ERROR] server-056c5c14-3305-18be-9824-f917cef0ab0e: failed to make requestVote RPC: target="{Voter server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 0f74db2c-ded7-6807-fe92-1bb9c00da2f8}" error="failed to connect to peer: 0f74db2c-ded7-6807-fe92-1bb9c00da2f8" testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.540Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.540Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.541Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering candidate state: node="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Candidate]" term=3 testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.541Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering candidate state: node="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Candidate]" term=3 testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.541Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to make requestVote RPC: target="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.541Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to make requestVote RPC: target="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.591Z [WARN] server-056c5c14-3305-18be-9824-f917cef0ab0e: Election timeout reached, restarting election testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.591Z [WARN] server-056c5c14-3305-18be-9824-f917cef0ab0e: Election timeout reached, restarting election testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.592Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: entering candidate state: node="Node at 056c5c14-3305-18be-9824-f917cef0ab0e [Candidate]" term=4 testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.592Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: entering candidate state: node="Node at 056c5c14-3305-18be-9824-f917cef0ab0e [Candidate]" term=4 testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.592Z [ERROR] server-056c5c14-3305-18be-9824-f917cef0ab0e: failed to make requestVote RPC: target="{Voter server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 0f74db2c-ded7-6807-fe92-1bb9c00da2f8}" error="failed to connect to peer: 0f74db2c-ded7-6807-fe92-1bb9c00da2f8" testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.592Z [ERROR] server-056c5c14-3305-18be-9824-f917cef0ab0e: failed to make requestVote RPC: target="{Voter server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 0f74db2c-ded7-6807-fe92-1bb9c00da2f8}" error="failed to connect to peer: 0f74db2c-ded7-6807-fe92-1bb9c00da2f8" testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.600Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: Election timeout reached, restarting election testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.600Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: Election timeout reached, restarting election testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.601Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering candidate state: node="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Candidate]" term=4 testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.601Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering candidate state: node="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Candidate]" term=4 testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.601Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to make requestVote RPC: target="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.601Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to make requestVote RPC: target="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.602Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.497597213 +0000 UTC m=+14.261813122 (0 nodes), 10.358597ms from start of poll, 367.823994ms from cluster start. Timeout at 2021-08-25 09:28:14.602643671 +0000 UTC m=+14.366859640, 105.046518ms after stability testing.go:149: cluster: 2021-08-25T09:28:14.602Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.497597213 +0000 UTC m=+14.261813122 (0 nodes), 10.358597ms from start of poll, 367.823994ms from cluster start. Timeout at 2021-08-25 09:28:14.602643671 +0000 UTC m=+14.366859640, 105.046518ms after stability testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.653Z [WARN] server-056c5c14-3305-18be-9824-f917cef0ab0e: Election timeout reached, restarting election testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.653Z [WARN] server-056c5c14-3305-18be-9824-f917cef0ab0e: Election timeout reached, restarting election testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.654Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: entering candidate state: node="Node at 056c5c14-3305-18be-9824-f917cef0ab0e [Candidate]" term=5 testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.654Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: entering candidate state: node="Node at 056c5c14-3305-18be-9824-f917cef0ab0e [Candidate]" term=5 testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.654Z [ERROR] server-056c5c14-3305-18be-9824-f917cef0ab0e: failed to make requestVote RPC: target="{Voter server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 0f74db2c-ded7-6807-fe92-1bb9c00da2f8}" error="failed to connect to peer: 0f74db2c-ded7-6807-fe92-1bb9c00da2f8" testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.654Z [ERROR] server-056c5c14-3305-18be-9824-f917cef0ab0e: failed to make requestVote RPC: target="{Voter server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 0f74db2c-ded7-6807-fe92-1bb9c00da2f8}" error="failed to connect to peer: 0f74db2c-ded7-6807-fe92-1bb9c00da2f8" testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.675Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: Election timeout reached, restarting election testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.675Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: Election timeout reached, restarting election testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.675Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering candidate state: node="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Candidate]" term=5 testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.675Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering candidate state: node="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Candidate]" term=5 testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.675Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to make requestVote RPC: target="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.675Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to make requestVote RPC: target="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" --- PASS: TestRaft_LeaderLeaseExpire (0.58s) === RUN TestRaft_Barrier testLoggerAdapter verbose: server-441b9051-90cb-b2b4-5a24-a088a275ab3e: 2021-08-25T09:28:14.711Z [INFO] server-441b9051-90cb-b2b4-5a24-a088a275ab3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-441b9051-90cb-b2b4-5a24-a088a275ab3e Address:441b9051-90cb-b2b4-5a24-a088a275ab3e} {Suffrage:Voter ID:server-4598815a-3bc5-63dc-d49a-21e462306ae0 Address:4598815a-3bc5-63dc-d49a-21e462306ae0} {Suffrage:Voter ID:server-c5e140b4-a14d-3987-b3b9-79f8927f619c Address:c5e140b4-a14d-3987-b3b9-79f8927f619c}]" testing.go:149: server-441b9051-90cb-b2b4-5a24-a088a275ab3e: 2021-08-25T09:28:14.711Z [INFO] server-441b9051-90cb-b2b4-5a24-a088a275ab3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-441b9051-90cb-b2b4-5a24-a088a275ab3e Address:441b9051-90cb-b2b4-5a24-a088a275ab3e} {Suffrage:Voter ID:server-4598815a-3bc5-63dc-d49a-21e462306ae0 Address:4598815a-3bc5-63dc-d49a-21e462306ae0} {Suffrage:Voter ID:server-c5e140b4-a14d-3987-b3b9-79f8927f619c Address:c5e140b4-a14d-3987-b3b9-79f8927f619c}]" testLoggerAdapter verbose: server-441b9051-90cb-b2b4-5a24-a088a275ab3e: 2021-08-25T09:28:14.712Z [INFO] server-441b9051-90cb-b2b4-5a24-a088a275ab3e: entering follower state: follower="Node at 441b9051-90cb-b2b4-5a24-a088a275ab3e [Follower]" leader= testing.go:149: server-441b9051-90cb-b2b4-5a24-a088a275ab3e: 2021-08-25T09:28:14.712Z [INFO] server-441b9051-90cb-b2b4-5a24-a088a275ab3e: entering follower state: follower="Node at 441b9051-90cb-b2b4-5a24-a088a275ab3e [Follower]" leader= testLoggerAdapter verbose: server-4598815a-3bc5-63dc-d49a-21e462306ae0: 2021-08-25T09:28:14.712Z [INFO] server-4598815a-3bc5-63dc-d49a-21e462306ae0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-441b9051-90cb-b2b4-5a24-a088a275ab3e Address:441b9051-90cb-b2b4-5a24-a088a275ab3e} {Suffrage:Voter ID:server-4598815a-3bc5-63dc-d49a-21e462306ae0 Address:4598815a-3bc5-63dc-d49a-21e462306ae0} {Suffrage:Voter ID:server-c5e140b4-a14d-3987-b3b9-79f8927f619c Address:c5e140b4-a14d-3987-b3b9-79f8927f619c}]" testing.go:149: server-4598815a-3bc5-63dc-d49a-21e462306ae0: 2021-08-25T09:28:14.712Z [INFO] server-4598815a-3bc5-63dc-d49a-21e462306ae0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-441b9051-90cb-b2b4-5a24-a088a275ab3e Address:441b9051-90cb-b2b4-5a24-a088a275ab3e} {Suffrage:Voter ID:server-4598815a-3bc5-63dc-d49a-21e462306ae0 Address:4598815a-3bc5-63dc-d49a-21e462306ae0} {Suffrage:Voter ID:server-c5e140b4-a14d-3987-b3b9-79f8927f619c Address:c5e140b4-a14d-3987-b3b9-79f8927f619c}]" testLoggerAdapter verbose: server-4598815a-3bc5-63dc-d49a-21e462306ae0: 2021-08-25T09:28:14.712Z [INFO] server-4598815a-3bc5-63dc-d49a-21e462306ae0: entering follower state: follower="Node at 4598815a-3bc5-63dc-d49a-21e462306ae0 [Follower]" leader= testing.go:149: server-4598815a-3bc5-63dc-d49a-21e462306ae0: 2021-08-25T09:28:14.712Z [INFO] server-4598815a-3bc5-63dc-d49a-21e462306ae0: entering follower state: follower="Node at 4598815a-3bc5-63dc-d49a-21e462306ae0 [Follower]" leader= testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.712Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-441b9051-90cb-b2b4-5a24-a088a275ab3e Address:441b9051-90cb-b2b4-5a24-a088a275ab3e} {Suffrage:Voter ID:server-4598815a-3bc5-63dc-d49a-21e462306ae0 Address:4598815a-3bc5-63dc-d49a-21e462306ae0} {Suffrage:Voter ID:server-c5e140b4-a14d-3987-b3b9-79f8927f619c Address:c5e140b4-a14d-3987-b3b9-79f8927f619c}]" testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.712Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-441b9051-90cb-b2b4-5a24-a088a275ab3e Address:441b9051-90cb-b2b4-5a24-a088a275ab3e} {Suffrage:Voter ID:server-4598815a-3bc5-63dc-d49a-21e462306ae0 Address:4598815a-3bc5-63dc-d49a-21e462306ae0} {Suffrage:Voter ID:server-c5e140b4-a14d-3987-b3b9-79f8927f619c Address:c5e140b4-a14d-3987-b3b9-79f8927f619c}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.712Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.713Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: entering follower state: follower="Node at c5e140b4-a14d-3987-b3b9-79f8927f619c [Follower]" leader= testing.go:149: cluster: 2021-08-25T09:28:14.712Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.713Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: entering follower state: follower="Node at c5e140b4-a14d-3987-b3b9-79f8927f619c [Follower]" leader= testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [WARN] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [WARN] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: entering candidate state: node="Node at c5e140b4-a14d-3987-b3b9-79f8927f619c [Candidate]" term=2 testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: entering candidate state: node="Node at c5e140b4-a14d-3987-b3b9-79f8927f619c [Candidate]" term=2 testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: election won: tally=2 testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: election won: tally=2 testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: entering leader state: leader="Node at c5e140b4-a14d-3987-b3b9-79f8927f619c [Leader]" testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: entering leader state: leader="Node at c5e140b4-a14d-3987-b3b9-79f8927f619c [Leader]" testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: added peer, starting replication: peer=server-441b9051-90cb-b2b4-5a24-a088a275ab3e testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: added peer, starting replication: peer=server-441b9051-90cb-b2b4-5a24-a088a275ab3e testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: added peer, starting replication: peer=server-4598815a-3bc5-63dc-d49a-21e462306ae0 testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: added peer, starting replication: peer=server-4598815a-3bc5-63dc-d49a-21e462306ae0 testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: pipelining replication: peer="{Voter server-441b9051-90cb-b2b4-5a24-a088a275ab3e 441b9051-90cb-b2b4-5a24-a088a275ab3e}" testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: pipelining replication: peer="{Voter server-441b9051-90cb-b2b4-5a24-a088a275ab3e 441b9051-90cb-b2b4-5a24-a088a275ab3e}" testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: pipelining replication: peer="{Voter server-4598815a-3bc5-63dc-d49a-21e462306ae0 4598815a-3bc5-63dc-d49a-21e462306ae0}" testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: pipelining replication: peer="{Voter server-4598815a-3bc5-63dc-d49a-21e462306ae0 4598815a-3bc5-63dc-d49a-21e462306ae0}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.879Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.774521856 +0000 UTC m=+14.538737745 (1 nodes), 61.43865ms from start of poll, 62.847699ms from cluster start. Timeout at 2021-08-25 09:28:14.879544316 +0000 UTC m=+14.643760245, 105.0225ms after stability testing.go:149: cluster: 2021-08-25T09:28:14.879Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.774521856 +0000 UTC m=+14.538737745 (1 nodes), 61.43865ms from start of poll, 62.847699ms from cluster start. Timeout at 2021-08-25 09:28:14.879544316 +0000 UTC m=+14.643760245, 105.0225ms after stability testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.894Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: aborting pipeline replication: peer="{Voter server-4598815a-3bc5-63dc-d49a-21e462306ae0 4598815a-3bc5-63dc-d49a-21e462306ae0}" testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.894Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: aborting pipeline replication: peer="{Voter server-4598815a-3bc5-63dc-d49a-21e462306ae0 4598815a-3bc5-63dc-d49a-21e462306ae0}" testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.894Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: aborting pipeline replication: peer="{Voter server-441b9051-90cb-b2b4-5a24-a088a275ab3e 441b9051-90cb-b2b4-5a24-a088a275ab3e}" testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.894Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: aborting pipeline replication: peer="{Voter server-441b9051-90cb-b2b4-5a24-a088a275ab3e 441b9051-90cb-b2b4-5a24-a088a275ab3e}" --- PASS: TestRaft_Barrier (0.19s) === RUN TestRaft_VerifyLeader testLoggerAdapter verbose: server-d77979d5-3e0c-27a6-5587-5e3136457df4: 2021-08-25T09:28:14.898Z [INFO] server-d77979d5-3e0c-27a6-5587-5e3136457df4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d77979d5-3e0c-27a6-5587-5e3136457df4 Address:d77979d5-3e0c-27a6-5587-5e3136457df4} {Suffrage:Voter ID:server-736cb78a-a94c-bf71-c6c9-9b33575daa09 Address:736cb78a-a94c-bf71-c6c9-9b33575daa09} {Suffrage:Voter ID:server-bf526228-a556-793d-c54e-8a62b64cc54b Address:bf526228-a556-793d-c54e-8a62b64cc54b}]" testing.go:149: server-d77979d5-3e0c-27a6-5587-5e3136457df4: 2021-08-25T09:28:14.898Z [INFO] server-d77979d5-3e0c-27a6-5587-5e3136457df4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d77979d5-3e0c-27a6-5587-5e3136457df4 Address:d77979d5-3e0c-27a6-5587-5e3136457df4} {Suffrage:Voter ID:server-736cb78a-a94c-bf71-c6c9-9b33575daa09 Address:736cb78a-a94c-bf71-c6c9-9b33575daa09} {Suffrage:Voter ID:server-bf526228-a556-793d-c54e-8a62b64cc54b Address:bf526228-a556-793d-c54e-8a62b64cc54b}]" testLoggerAdapter verbose: server-d77979d5-3e0c-27a6-5587-5e3136457df4: 2021-08-25T09:28:14.898Z [INFO] server-d77979d5-3e0c-27a6-5587-5e3136457df4: entering follower state: follower="Node at d77979d5-3e0c-27a6-5587-5e3136457df4 [Follower]" leader= testing.go:149: server-d77979d5-3e0c-27a6-5587-5e3136457df4: 2021-08-25T09:28:14.898Z [INFO] server-d77979d5-3e0c-27a6-5587-5e3136457df4: entering follower state: follower="Node at d77979d5-3e0c-27a6-5587-5e3136457df4 [Follower]" leader= testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.899Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d77979d5-3e0c-27a6-5587-5e3136457df4 Address:d77979d5-3e0c-27a6-5587-5e3136457df4} {Suffrage:Voter ID:server-736cb78a-a94c-bf71-c6c9-9b33575daa09 Address:736cb78a-a94c-bf71-c6c9-9b33575daa09} {Suffrage:Voter ID:server-bf526228-a556-793d-c54e-8a62b64cc54b Address:bf526228-a556-793d-c54e-8a62b64cc54b}]" testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.899Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d77979d5-3e0c-27a6-5587-5e3136457df4 Address:d77979d5-3e0c-27a6-5587-5e3136457df4} {Suffrage:Voter ID:server-736cb78a-a94c-bf71-c6c9-9b33575daa09 Address:736cb78a-a94c-bf71-c6c9-9b33575daa09} {Suffrage:Voter ID:server-bf526228-a556-793d-c54e-8a62b64cc54b Address:bf526228-a556-793d-c54e-8a62b64cc54b}]" testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.899Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: entering follower state: follower="Node at 736cb78a-a94c-bf71-c6c9-9b33575daa09 [Follower]" leader= testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.899Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: entering follower state: follower="Node at 736cb78a-a94c-bf71-c6c9-9b33575daa09 [Follower]" leader= testLoggerAdapter verbose: server-bf526228-a556-793d-c54e-8a62b64cc54b: 2021-08-25T09:28:14.899Z [INFO] server-bf526228-a556-793d-c54e-8a62b64cc54b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d77979d5-3e0c-27a6-5587-5e3136457df4 Address:d77979d5-3e0c-27a6-5587-5e3136457df4} {Suffrage:Voter ID:server-736cb78a-a94c-bf71-c6c9-9b33575daa09 Address:736cb78a-a94c-bf71-c6c9-9b33575daa09} {Suffrage:Voter ID:server-bf526228-a556-793d-c54e-8a62b64cc54b Address:bf526228-a556-793d-c54e-8a62b64cc54b}]" testing.go:149: server-bf526228-a556-793d-c54e-8a62b64cc54b: 2021-08-25T09:28:14.899Z [INFO] server-bf526228-a556-793d-c54e-8a62b64cc54b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d77979d5-3e0c-27a6-5587-5e3136457df4 Address:d77979d5-3e0c-27a6-5587-5e3136457df4} {Suffrage:Voter ID:server-736cb78a-a94c-bf71-c6c9-9b33575daa09 Address:736cb78a-a94c-bf71-c6c9-9b33575daa09} {Suffrage:Voter ID:server-bf526228-a556-793d-c54e-8a62b64cc54b Address:bf526228-a556-793d-c54e-8a62b64cc54b}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.899Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:14.899Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-bf526228-a556-793d-c54e-8a62b64cc54b: 2021-08-25T09:28:14.899Z [INFO] server-bf526228-a556-793d-c54e-8a62b64cc54b: entering follower state: follower="Node at bf526228-a556-793d-c54e-8a62b64cc54b [Follower]" leader= testing.go:149: server-bf526228-a556-793d-c54e-8a62b64cc54b: 2021-08-25T09:28:14.899Z [INFO] server-bf526228-a556-793d-c54e-8a62b64cc54b: entering follower state: follower="Node at bf526228-a556-793d-c54e-8a62b64cc54b [Follower]" leader= testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [WARN] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [WARN] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: entering candidate state: node="Node at 736cb78a-a94c-bf71-c6c9-9b33575daa09 [Candidate]" term=2 testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: entering candidate state: node="Node at 736cb78a-a94c-bf71-c6c9-9b33575daa09 [Candidate]" term=2 testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: election won: tally=2 testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: election won: tally=2 testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: entering leader state: leader="Node at 736cb78a-a94c-bf71-c6c9-9b33575daa09 [Leader]" testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: entering leader state: leader="Node at 736cb78a-a94c-bf71-c6c9-9b33575daa09 [Leader]" testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: added peer, starting replication: peer=server-d77979d5-3e0c-27a6-5587-5e3136457df4 testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: added peer, starting replication: peer=server-d77979d5-3e0c-27a6-5587-5e3136457df4 testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: added peer, starting replication: peer=server-bf526228-a556-793d-c54e-8a62b64cc54b testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: added peer, starting replication: peer=server-bf526228-a556-793d-c54e-8a62b64cc54b testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: pipelining replication: peer="{Voter server-d77979d5-3e0c-27a6-5587-5e3136457df4 d77979d5-3e0c-27a6-5587-5e3136457df4}" testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: pipelining replication: peer="{Voter server-d77979d5-3e0c-27a6-5587-5e3136457df4 d77979d5-3e0c-27a6-5587-5e3136457df4}" testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: pipelining replication: peer="{Voter server-bf526228-a556-793d-c54e-8a62b64cc54b bf526228-a556-793d-c54e-8a62b64cc54b}" testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: pipelining replication: peer="{Voter server-bf526228-a556-793d-c54e-8a62b64cc54b bf526228-a556-793d-c54e-8a62b64cc54b}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.056Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.951724287 +0000 UTC m=+14.715940156 (1 nodes), 51.959671ms from start of poll, 53.233292ms from cluster start. Timeout at 2021-08-25 09:28:15.056841698 +0000 UTC m=+14.821057647, 105.117491ms after stability testing.go:149: cluster: 2021-08-25T09:28:15.056Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.951724287 +0000 UTC m=+14.715940156 (1 nodes), 51.959671ms from start of poll, 53.233292ms from cluster start. Timeout at 2021-08-25 09:28:15.056841698 +0000 UTC m=+14.821057647, 105.117491ms after stability testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:15.057Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: aborting pipeline replication: peer="{Voter server-bf526228-a556-793d-c54e-8a62b64cc54b bf526228-a556-793d-c54e-8a62b64cc54b}" testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:15.057Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: aborting pipeline replication: peer="{Voter server-bf526228-a556-793d-c54e-8a62b64cc54b bf526228-a556-793d-c54e-8a62b64cc54b}" testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:15.057Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: aborting pipeline replication: peer="{Voter server-d77979d5-3e0c-27a6-5587-5e3136457df4 d77979d5-3e0c-27a6-5587-5e3136457df4}" testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:15.057Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: aborting pipeline replication: peer="{Voter server-d77979d5-3e0c-27a6-5587-5e3136457df4 d77979d5-3e0c-27a6-5587-5e3136457df4}" --- PASS: TestRaft_VerifyLeader (0.16s) === RUN TestRaft_VerifyLeader_Single testLoggerAdapter verbose: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.060Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-466af8da-ea3c-48eb-2e8c-965c4b963578 Address:466af8da-ea3c-48eb-2e8c-965c4b963578}]" testing.go:149: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.060Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-466af8da-ea3c-48eb-2e8c-965c4b963578 Address:466af8da-ea3c-48eb-2e8c-965c4b963578}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.060Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:15.060Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.060Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: entering follower state: follower="Node at 466af8da-ea3c-48eb-2e8c-965c4b963578 [Follower]" leader= testing.go:149: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.060Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: entering follower state: follower="Node at 466af8da-ea3c-48eb-2e8c-965c4b963578 [Follower]" leader= testLoggerAdapter verbose: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.134Z [WARN] server-466af8da-ea3c-48eb-2e8c-965c4b963578: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.134Z [WARN] server-466af8da-ea3c-48eb-2e8c-965c4b963578: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.134Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: entering candidate state: node="Node at 466af8da-ea3c-48eb-2e8c-965c4b963578 [Candidate]" term=2 testing.go:149: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.134Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: entering candidate state: node="Node at 466af8da-ea3c-48eb-2e8c-965c4b963578 [Candidate]" term=2 testLoggerAdapter verbose: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.134Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: election won: tally=1 testing.go:149: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.134Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: election won: tally=1 testLoggerAdapter verbose: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.134Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: entering leader state: leader="Node at 466af8da-ea3c-48eb-2e8c-965c4b963578 [Leader]" testing.go:149: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.134Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: entering leader state: leader="Node at 466af8da-ea3c-48eb-2e8c-965c4b963578 [Leader]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.240Z [INFO] stable state for Leader reached at 2021-08-25 09:28:15.134869726 +0000 UTC m=+14.899085615 (1 nodes), 74.666941ms from start of poll, 75.095361ms from cluster start. Timeout at 2021-08-25 09:28:15.240013934 +0000 UTC m=+15.004229863, 105.144248ms after stability testing.go:149: cluster: 2021-08-25T09:28:15.240Z [INFO] stable state for Leader reached at 2021-08-25 09:28:15.134869726 +0000 UTC m=+14.899085615 (1 nodes), 74.666941ms from start of poll, 75.095361ms from cluster start. Timeout at 2021-08-25 09:28:15.240013934 +0000 UTC m=+15.004229863, 105.144248ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.18s) === RUN TestRaft_VerifyLeader_Fail testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.242Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-869a35fe-e12f-49f8-8547-bdfd07dfd703 Address:869a35fe-e12f-49f8-8547-bdfd07dfd703} {Suffrage:Voter ID:server-cc036e16-8377-4c65-99df-f407746bc6a7 Address:cc036e16-8377-4c65-99df-f407746bc6a7}]" testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.242Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-869a35fe-e12f-49f8-8547-bdfd07dfd703 Address:869a35fe-e12f-49f8-8547-bdfd07dfd703} {Suffrage:Voter ID:server-cc036e16-8377-4c65-99df-f407746bc6a7 Address:cc036e16-8377-4c65-99df-f407746bc6a7}]" testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.242Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: entering follower state: follower="Node at 869a35fe-e12f-49f8-8547-bdfd07dfd703 [Follower]" leader= testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.242Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: entering follower state: follower="Node at 869a35fe-e12f-49f8-8547-bdfd07dfd703 [Follower]" leader= testLoggerAdapter verbose: server-cc036e16-8377-4c65-99df-f407746bc6a7: 2021-08-25T09:28:15.243Z [INFO] server-cc036e16-8377-4c65-99df-f407746bc6a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-869a35fe-e12f-49f8-8547-bdfd07dfd703 Address:869a35fe-e12f-49f8-8547-bdfd07dfd703} {Suffrage:Voter ID:server-cc036e16-8377-4c65-99df-f407746bc6a7 Address:cc036e16-8377-4c65-99df-f407746bc6a7}]" testing.go:149: server-cc036e16-8377-4c65-99df-f407746bc6a7: 2021-08-25T09:28:15.243Z [INFO] server-cc036e16-8377-4c65-99df-f407746bc6a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-869a35fe-e12f-49f8-8547-bdfd07dfd703 Address:869a35fe-e12f-49f8-8547-bdfd07dfd703} {Suffrage:Voter ID:server-cc036e16-8377-4c65-99df-f407746bc6a7 Address:cc036e16-8377-4c65-99df-f407746bc6a7}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.243Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:15.243Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-cc036e16-8377-4c65-99df-f407746bc6a7: 2021-08-25T09:28:15.243Z [INFO] server-cc036e16-8377-4c65-99df-f407746bc6a7: entering follower state: follower="Node at cc036e16-8377-4c65-99df-f407746bc6a7 [Follower]" leader= testing.go:149: server-cc036e16-8377-4c65-99df-f407746bc6a7: 2021-08-25T09:28:15.243Z [INFO] server-cc036e16-8377-4c65-99df-f407746bc6a7: entering follower state: follower="Node at cc036e16-8377-4c65-99df-f407746bc6a7 [Follower]" leader= testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.299Z [WARN] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.299Z [WARN] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.299Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: entering candidate state: node="Node at 869a35fe-e12f-49f8-8547-bdfd07dfd703 [Candidate]" term=2 testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.299Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: entering candidate state: node="Node at 869a35fe-e12f-49f8-8547-bdfd07dfd703 [Candidate]" term=2 testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.300Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: election won: tally=2 testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.300Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: election won: tally=2 testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.300Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: entering leader state: leader="Node at 869a35fe-e12f-49f8-8547-bdfd07dfd703 [Leader]" testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.300Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: entering leader state: leader="Node at 869a35fe-e12f-49f8-8547-bdfd07dfd703 [Leader]" testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.300Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: added peer, starting replication: peer=server-cc036e16-8377-4c65-99df-f407746bc6a7 testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.300Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: added peer, starting replication: peer=server-cc036e16-8377-4c65-99df-f407746bc6a7 testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.300Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: pipelining replication: peer="{Voter server-cc036e16-8377-4c65-99df-f407746bc6a7 cc036e16-8377-4c65-99df-f407746bc6a7}" testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.300Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: pipelining replication: peer="{Voter server-cc036e16-8377-4c65-99df-f407746bc6a7 cc036e16-8377-4c65-99df-f407746bc6a7}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.405Z [INFO] stable state for Leader reached at 2021-08-25 09:28:15.300281893 +0000 UTC m=+15.064497802 (1 nodes), 57.071696ms from start of poll, 57.906239ms from cluster start. Timeout at 2021-08-25 09:28:15.405409982 +0000 UTC m=+15.169625911, 105.128109ms after stability testing.go:149: cluster: 2021-08-25T09:28:15.405Z [INFO] stable state for Leader reached at 2021-08-25 09:28:15.300281893 +0000 UTC m=+15.064497802 (1 nodes), 57.071696ms from start of poll, 57.906239ms from cluster start. Timeout at 2021-08-25 09:28:15.405409982 +0000 UTC m=+15.169625911, 105.128109ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.405Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:15.405Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.510Z [INFO] stable state for Follower reached at 2021-08-25 09:28:15.405696135 +0000 UTC m=+15.169912024 (1 nodes), 2.159µs from start of poll, 163.320461ms from cluster start. Timeout at 2021-08-25 09:28:15.510831224 +0000 UTC m=+15.275047253, 105.135229ms after stability testing.go:149: cluster: 2021-08-25T09:28:15.510Z [INFO] stable state for Follower reached at 2021-08-25 09:28:15.405696135 +0000 UTC m=+15.169912024 (1 nodes), 2.159µs from start of poll, 163.320461ms from cluster start. Timeout at 2021-08-25 09:28:15.510831224 +0000 UTC m=+15.275047253, 105.135229ms after stability testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.511Z [WARN] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: new leader elected, stepping down testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.511Z [WARN] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: new leader elected, stepping down testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.511Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: entering follower state: follower="Node at 869a35fe-e12f-49f8-8547-bdfd07dfd703 [Follower]" leader= testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.511Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: entering follower state: follower="Node at 869a35fe-e12f-49f8-8547-bdfd07dfd703 [Follower]" leader= testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.511Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: aborting pipeline replication: peer="{Voter server-cc036e16-8377-4c65-99df-f407746bc6a7 cc036e16-8377-4c65-99df-f407746bc6a7}" testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.511Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: aborting pipeline replication: peer="{Voter server-cc036e16-8377-4c65-99df-f407746bc6a7 cc036e16-8377-4c65-99df-f407746bc6a7}" --- PASS: TestRaft_VerifyLeader_Fail (0.27s) === RUN TestRaft_VerifyLeader_PartialConnect testLoggerAdapter verbose: server-9ce58e74-9aac-7f51-6be4-deecc99ceacd: 2021-08-25T09:28:15.514Z [INFO] server-9ce58e74-9aac-7f51-6be4-deecc99ceacd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ce58e74-9aac-7f51-6be4-deecc99ceacd Address:9ce58e74-9aac-7f51-6be4-deecc99ceacd} {Suffrage:Voter ID:server-507aad92-91f2-0b08-b12f-363dec5e687a Address:507aad92-91f2-0b08-b12f-363dec5e687a} {Suffrage:Voter ID:server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7 Address:1ef90d23-f193-2c79-49e9-a0c7a27be6d7}]" testing.go:149: server-9ce58e74-9aac-7f51-6be4-deecc99ceacd: 2021-08-25T09:28:15.514Z [INFO] server-9ce58e74-9aac-7f51-6be4-deecc99ceacd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ce58e74-9aac-7f51-6be4-deecc99ceacd Address:9ce58e74-9aac-7f51-6be4-deecc99ceacd} {Suffrage:Voter ID:server-507aad92-91f2-0b08-b12f-363dec5e687a Address:507aad92-91f2-0b08-b12f-363dec5e687a} {Suffrage:Voter ID:server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7 Address:1ef90d23-f193-2c79-49e9-a0c7a27be6d7}]" testLoggerAdapter verbose: server-9ce58e74-9aac-7f51-6be4-deecc99ceacd: 2021-08-25T09:28:15.514Z [INFO] server-9ce58e74-9aac-7f51-6be4-deecc99ceacd: entering follower state: follower="Node at 9ce58e74-9aac-7f51-6be4-deecc99ceacd [Follower]" leader= testing.go:149: server-9ce58e74-9aac-7f51-6be4-deecc99ceacd: 2021-08-25T09:28:15.514Z [INFO] server-9ce58e74-9aac-7f51-6be4-deecc99ceacd: entering follower state: follower="Node at 9ce58e74-9aac-7f51-6be4-deecc99ceacd [Follower]" leader= testLoggerAdapter verbose: server-507aad92-91f2-0b08-b12f-363dec5e687a: 2021-08-25T09:28:15.514Z [INFO] server-507aad92-91f2-0b08-b12f-363dec5e687a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ce58e74-9aac-7f51-6be4-deecc99ceacd Address:9ce58e74-9aac-7f51-6be4-deecc99ceacd} {Suffrage:Voter ID:server-507aad92-91f2-0b08-b12f-363dec5e687a Address:507aad92-91f2-0b08-b12f-363dec5e687a} {Suffrage:Voter ID:server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7 Address:1ef90d23-f193-2c79-49e9-a0c7a27be6d7}]" testing.go:149: server-507aad92-91f2-0b08-b12f-363dec5e687a: 2021-08-25T09:28:15.514Z [INFO] server-507aad92-91f2-0b08-b12f-363dec5e687a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ce58e74-9aac-7f51-6be4-deecc99ceacd Address:9ce58e74-9aac-7f51-6be4-deecc99ceacd} {Suffrage:Voter ID:server-507aad92-91f2-0b08-b12f-363dec5e687a Address:507aad92-91f2-0b08-b12f-363dec5e687a} {Suffrage:Voter ID:server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7 Address:1ef90d23-f193-2c79-49e9-a0c7a27be6d7}]" testLoggerAdapter verbose: server-507aad92-91f2-0b08-b12f-363dec5e687a: 2021-08-25T09:28:15.514Z [INFO] server-507aad92-91f2-0b08-b12f-363dec5e687a: entering follower state: follower="Node at 507aad92-91f2-0b08-b12f-363dec5e687a [Follower]" leader= testing.go:149: server-507aad92-91f2-0b08-b12f-363dec5e687a: 2021-08-25T09:28:15.514Z [INFO] server-507aad92-91f2-0b08-b12f-363dec5e687a: entering follower state: follower="Node at 507aad92-91f2-0b08-b12f-363dec5e687a [Follower]" leader= testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.515Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ce58e74-9aac-7f51-6be4-deecc99ceacd Address:9ce58e74-9aac-7f51-6be4-deecc99ceacd} {Suffrage:Voter ID:server-507aad92-91f2-0b08-b12f-363dec5e687a Address:507aad92-91f2-0b08-b12f-363dec5e687a} {Suffrage:Voter ID:server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7 Address:1ef90d23-f193-2c79-49e9-a0c7a27be6d7}]" testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.515Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ce58e74-9aac-7f51-6be4-deecc99ceacd Address:9ce58e74-9aac-7f51-6be4-deecc99ceacd} {Suffrage:Voter ID:server-507aad92-91f2-0b08-b12f-363dec5e687a Address:507aad92-91f2-0b08-b12f-363dec5e687a} {Suffrage:Voter ID:server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7 Address:1ef90d23-f193-2c79-49e9-a0c7a27be6d7}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.515Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.515Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: entering follower state: follower="Node at 1ef90d23-f193-2c79-49e9-a0c7a27be6d7 [Follower]" leader= testing.go:149: cluster: 2021-08-25T09:28:15.515Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.515Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: entering follower state: follower="Node at 1ef90d23-f193-2c79-49e9-a0c7a27be6d7 [Follower]" leader= testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [WARN] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [WARN] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: entering candidate state: node="Node at 1ef90d23-f193-2c79-49e9-a0c7a27be6d7 [Candidate]" term=2 testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: entering candidate state: node="Node at 1ef90d23-f193-2c79-49e9-a0c7a27be6d7 [Candidate]" term=2 testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: election won: tally=2 testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: election won: tally=2 testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: entering leader state: leader="Node at 1ef90d23-f193-2c79-49e9-a0c7a27be6d7 [Leader]" testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: entering leader state: leader="Node at 1ef90d23-f193-2c79-49e9-a0c7a27be6d7 [Leader]" testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: added peer, starting replication: peer=server-9ce58e74-9aac-7f51-6be4-deecc99ceacd testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: added peer, starting replication: peer=server-9ce58e74-9aac-7f51-6be4-deecc99ceacd testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: added peer, starting replication: peer=server-507aad92-91f2-0b08-b12f-363dec5e687a testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: added peer, starting replication: peer=server-507aad92-91f2-0b08-b12f-363dec5e687a testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: pipelining replication: peer="{Voter server-507aad92-91f2-0b08-b12f-363dec5e687a 507aad92-91f2-0b08-b12f-363dec5e687a}" testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: pipelining replication: peer="{Voter server-507aad92-91f2-0b08-b12f-363dec5e687a 507aad92-91f2-0b08-b12f-363dec5e687a}" testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.581Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: pipelining replication: peer="{Voter server-9ce58e74-9aac-7f51-6be4-deecc99ceacd 9ce58e74-9aac-7f51-6be4-deecc99ceacd}" testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.581Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: pipelining replication: peer="{Voter server-9ce58e74-9aac-7f51-6be4-deecc99ceacd 9ce58e74-9aac-7f51-6be4-deecc99ceacd}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.686Z [INFO] stable state for Leader reached at 2021-08-25 09:28:15.581369988 +0000 UTC m=+15.345585897 (1 nodes), 65.95033ms from start of poll, 67.217292ms from cluster start. Timeout at 2021-08-25 09:28:15.686557272 +0000 UTC m=+15.450773261, 105.187364ms after stability testing.go:149: cluster: 2021-08-25T09:28:15.686Z [INFO] stable state for Leader reached at 2021-08-25 09:28:15.581369988 +0000 UTC m=+15.345585897 (1 nodes), 65.95033ms from start of poll, 67.217292ms from cluster start. Timeout at 2021-08-25 09:28:15.686557272 +0000 UTC m=+15.450773261, 105.187364ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.692Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:15.692Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.797Z [INFO] stable state for Follower reached at 2021-08-25 09:28:15.692155332 +0000 UTC m=+15.456371221 (2 nodes), 2.86µs from start of poll, 178.002616ms from cluster start. Timeout at 2021-08-25 09:28:15.797314099 +0000 UTC m=+15.561530048, 105.158827ms after stability testing.go:149: cluster: 2021-08-25T09:28:15.797Z [INFO] stable state for Follower reached at 2021-08-25 09:28:15.692155332 +0000 UTC m=+15.456371221 (2 nodes), 2.86µs from start of poll, 178.002616ms from cluster start. Timeout at 2021-08-25 09:28:15.797314099 +0000 UTC m=+15.561530048, 105.158827ms after stability raft_test.go:1647: [INFO] Disconnecting Node at 9ce58e74-9aac-7f51-6be4-deecc99ceacd [Follower] testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.797Z [ERROR] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: failed to heartbeat to: peer=9ce58e74-9aac-7f51-6be4-deecc99ceacd error="failed to connect to peer: 9ce58e74-9aac-7f51-6be4-deecc99ceacd" testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.797Z [ERROR] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: failed to heartbeat to: peer=9ce58e74-9aac-7f51-6be4-deecc99ceacd error="failed to connect to peer: 9ce58e74-9aac-7f51-6be4-deecc99ceacd" testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.797Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: aborting pipeline replication: peer="{Voter server-507aad92-91f2-0b08-b12f-363dec5e687a 507aad92-91f2-0b08-b12f-363dec5e687a}" testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.797Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: aborting pipeline replication: peer="{Voter server-507aad92-91f2-0b08-b12f-363dec5e687a 507aad92-91f2-0b08-b12f-363dec5e687a}" testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.797Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: aborting pipeline replication: peer="{Voter server-9ce58e74-9aac-7f51-6be4-deecc99ceacd 9ce58e74-9aac-7f51-6be4-deecc99ceacd}" testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.797Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: aborting pipeline replication: peer="{Voter server-9ce58e74-9aac-7f51-6be4-deecc99ceacd 9ce58e74-9aac-7f51-6be4-deecc99ceacd}" --- PASS: TestRaft_VerifyLeader_PartialConnect (0.29s) === RUN TestRaft_StartAsLeader testLoggerAdapter verbose: server-4459b3a1-3046-e4a6-ea59-ba718b18f246: 2021-08-25T09:28:15.800Z [INFO] server-4459b3a1-3046-e4a6-ea59-ba718b18f246: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4459b3a1-3046-e4a6-ea59-ba718b18f246 Address:4459b3a1-3046-e4a6-ea59-ba718b18f246}]" testing.go:149: server-4459b3a1-3046-e4a6-ea59-ba718b18f246: 2021-08-25T09:28:15.800Z [INFO] server-4459b3a1-3046-e4a6-ea59-ba718b18f246: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4459b3a1-3046-e4a6-ea59-ba718b18f246 Address:4459b3a1-3046-e4a6-ea59-ba718b18f246}]" testLoggerAdapter verbose: server-4459b3a1-3046-e4a6-ea59-ba718b18f246: 2021-08-25T09:28:15.801Z [INFO] server-4459b3a1-3046-e4a6-ea59-ba718b18f246: entering leader state: leader="Node at 4459b3a1-3046-e4a6-ea59-ba718b18f246 [Leader]" testing.go:149: server-4459b3a1-3046-e4a6-ea59-ba718b18f246: 2021-08-25T09:28:15.801Z [INFO] server-4459b3a1-3046-e4a6-ea59-ba718b18f246: entering leader state: leader="Node at 4459b3a1-3046-e4a6-ea59-ba718b18f246 [Leader]" --- PASS: TestRaft_StartAsLeader (0.00s) === RUN TestRaft_NotifyCh testLoggerAdapter verbose: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.803Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf Address:f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf}]" testing.go:149: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.803Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf Address:f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf}]" testLoggerAdapter verbose: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.803Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: entering follower state: follower="Node at f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf [Follower]" leader= testing.go:149: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.803Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: entering follower state: follower="Node at f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf [Follower]" leader= testLoggerAdapter verbose: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.869Z [WARN] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.869Z [WARN] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.869Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: entering candidate state: node="Node at f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf [Candidate]" term=2 testing.go:149: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.869Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: entering candidate state: node="Node at f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf [Candidate]" term=2 testLoggerAdapter verbose: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.869Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: election won: tally=1 testing.go:149: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.869Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: election won: tally=1 testLoggerAdapter verbose: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.869Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: entering leader state: leader="Node at f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf [Leader]" testing.go:149: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.869Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: entering leader state: leader="Node at f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf [Leader]" --- PASS: TestRaft_NotifyCh (0.07s) === RUN TestRaft_Voting testLoggerAdapter verbose: server-2bd15992-4d28-455a-9c76-9735991c0cb2: 2021-08-25T09:28:15.873Z [INFO] server-2bd15992-4d28-455a-9c76-9735991c0cb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2bd15992-4d28-455a-9c76-9735991c0cb2 Address:2bd15992-4d28-455a-9c76-9735991c0cb2} {Suffrage:Voter ID:server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 Address:7000105f-a0d1-5762-c2b6-75a830a9c2e9} {Suffrage:Voter ID:server-3bdcbafe-2ba8-d631-15c0-c8c80189a015 Address:3bdcbafe-2ba8-d631-15c0-c8c80189a015}]" testing.go:149: server-2bd15992-4d28-455a-9c76-9735991c0cb2: 2021-08-25T09:28:15.873Z [INFO] server-2bd15992-4d28-455a-9c76-9735991c0cb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2bd15992-4d28-455a-9c76-9735991c0cb2 Address:2bd15992-4d28-455a-9c76-9735991c0cb2} {Suffrage:Voter ID:server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 Address:7000105f-a0d1-5762-c2b6-75a830a9c2e9} {Suffrage:Voter ID:server-3bdcbafe-2ba8-d631-15c0-c8c80189a015 Address:3bdcbafe-2ba8-d631-15c0-c8c80189a015}]" testLoggerAdapter verbose: server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: 2021-08-25T09:28:15.874Z [INFO] server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2bd15992-4d28-455a-9c76-9735991c0cb2 Address:2bd15992-4d28-455a-9c76-9735991c0cb2} {Suffrage:Voter ID:server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 Address:7000105f-a0d1-5762-c2b6-75a830a9c2e9} {Suffrage:Voter ID:server-3bdcbafe-2ba8-d631-15c0-c8c80189a015 Address:3bdcbafe-2ba8-d631-15c0-c8c80189a015}]" testing.go:149: server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: 2021-08-25T09:28:15.874Z [INFO] server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2bd15992-4d28-455a-9c76-9735991c0cb2 Address:2bd15992-4d28-455a-9c76-9735991c0cb2} {Suffrage:Voter ID:server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 Address:7000105f-a0d1-5762-c2b6-75a830a9c2e9} {Suffrage:Voter ID:server-3bdcbafe-2ba8-d631-15c0-c8c80189a015 Address:3bdcbafe-2ba8-d631-15c0-c8c80189a015}]" testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.874Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2bd15992-4d28-455a-9c76-9735991c0cb2 Address:2bd15992-4d28-455a-9c76-9735991c0cb2} {Suffrage:Voter ID:server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 Address:7000105f-a0d1-5762-c2b6-75a830a9c2e9} {Suffrage:Voter ID:server-3bdcbafe-2ba8-d631-15c0-c8c80189a015 Address:3bdcbafe-2ba8-d631-15c0-c8c80189a015}]" testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.874Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2bd15992-4d28-455a-9c76-9735991c0cb2 Address:2bd15992-4d28-455a-9c76-9735991c0cb2} {Suffrage:Voter ID:server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 Address:7000105f-a0d1-5762-c2b6-75a830a9c2e9} {Suffrage:Voter ID:server-3bdcbafe-2ba8-d631-15c0-c8c80189a015 Address:3bdcbafe-2ba8-d631-15c0-c8c80189a015}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.875Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:15.875Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: 2021-08-25T09:28:15.875Z [INFO] server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: entering follower state: follower="Node at 7000105f-a0d1-5762-c2b6-75a830a9c2e9 [Follower]" leader= testing.go:149: server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: 2021-08-25T09:28:15.875Z [INFO] server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: entering follower state: follower="Node at 7000105f-a0d1-5762-c2b6-75a830a9c2e9 [Follower]" leader= testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.875Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: entering follower state: follower="Node at 3bdcbafe-2ba8-d631-15c0-c8c80189a015 [Follower]" leader= testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.875Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: entering follower state: follower="Node at 3bdcbafe-2ba8-d631-15c0-c8c80189a015 [Follower]" leader= testLoggerAdapter verbose: server-2bd15992-4d28-455a-9c76-9735991c0cb2: 2021-08-25T09:28:15.875Z [INFO] server-2bd15992-4d28-455a-9c76-9735991c0cb2: entering follower state: follower="Node at 2bd15992-4d28-455a-9c76-9735991c0cb2 [Follower]" leader= testing.go:149: server-2bd15992-4d28-455a-9c76-9735991c0cb2: 2021-08-25T09:28:15.875Z [INFO] server-2bd15992-4d28-455a-9c76-9735991c0cb2: entering follower state: follower="Node at 2bd15992-4d28-455a-9c76-9735991c0cb2 [Follower]" leader= testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.966Z [WARN] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.966Z [WARN] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.966Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: entering candidate state: node="Node at 3bdcbafe-2ba8-d631-15c0-c8c80189a015 [Candidate]" term=2 testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.966Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: entering candidate state: node="Node at 3bdcbafe-2ba8-d631-15c0-c8c80189a015 [Candidate]" term=2 testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.966Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: election won: tally=2 testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.966Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: election won: tally=2 testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: entering leader state: leader="Node at 3bdcbafe-2ba8-d631-15c0-c8c80189a015 [Leader]" testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: entering leader state: leader="Node at 3bdcbafe-2ba8-d631-15c0-c8c80189a015 [Leader]" testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: added peer, starting replication: peer=server-2bd15992-4d28-455a-9c76-9735991c0cb2 testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: added peer, starting replication: peer=server-2bd15992-4d28-455a-9c76-9735991c0cb2 testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: added peer, starting replication: peer=server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: added peer, starting replication: peer=server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: pipelining replication: peer="{Voter server-2bd15992-4d28-455a-9c76-9735991c0cb2 2bd15992-4d28-455a-9c76-9735991c0cb2}" testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: pipelining replication: peer="{Voter server-2bd15992-4d28-455a-9c76-9735991c0cb2 2bd15992-4d28-455a-9c76-9735991c0cb2}" testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: pipelining replication: peer="{Voter server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 7000105f-a0d1-5762-c2b6-75a830a9c2e9}" testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: pipelining replication: peer="{Voter server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 7000105f-a0d1-5762-c2b6-75a830a9c2e9}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.072Z [INFO] stable state for Follower reached at 2021-08-25 09:28:15.967036825 +0000 UTC m=+15.731252714 (2 nodes), 91.963053ms from start of poll, 93.557905ms from cluster start. Timeout at 2021-08-25 09:28:16.072180793 +0000 UTC m=+15.836396702, 105.143988ms after stability testing.go:149: cluster: 2021-08-25T09:28:16.072Z [INFO] stable state for Follower reached at 2021-08-25 09:28:15.967036825 +0000 UTC m=+15.731252714 (2 nodes), 91.963053ms from start of poll, 93.557905ms from cluster start. Timeout at 2021-08-25 09:28:16.072180793 +0000 UTC m=+15.836396702, 105.143988ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.072Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:16.072Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.177Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.072475565 +0000 UTC m=+15.836691454 (1 nodes), 2.799µs from start of poll, 198.996645ms from cluster start. Timeout at 2021-08-25 09:28:16.177642351 +0000 UTC m=+15.941858300, 105.166846ms after stability testing.go:149: cluster: 2021-08-25T09:28:16.177Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.072475565 +0000 UTC m=+15.836691454 (1 nodes), 2.799µs from start of poll, 198.996645ms from cluster start. Timeout at 2021-08-25 09:28:16.177642351 +0000 UTC m=+15.941858300, 105.166846ms after stability testLoggerAdapter verbose: server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: 2021-08-25T09:28:16.177Z [WARN] server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: rejecting vote request since we have a leader: from=2bd15992-4d28-455a-9c76-9735991c0cb2 leader=3bdcbafe-2ba8-d631-15c0-c8c80189a015 testing.go:149: server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: 2021-08-25T09:28:16.177Z [WARN] server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: rejecting vote request since we have a leader: from=2bd15992-4d28-455a-9c76-9735991c0cb2 leader=3bdcbafe-2ba8-d631-15c0-c8c80189a015 testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:16.178Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: aborting pipeline replication: peer="{Voter server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 7000105f-a0d1-5762-c2b6-75a830a9c2e9}" testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:16.178Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: aborting pipeline replication: peer="{Voter server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 7000105f-a0d1-5762-c2b6-75a830a9c2e9}" testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:16.178Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: aborting pipeline replication: peer="{Voter server-2bd15992-4d28-455a-9c76-9735991c0cb2 2bd15992-4d28-455a-9c76-9735991c0cb2}" testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:16.178Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: aborting pipeline replication: peer="{Voter server-2bd15992-4d28-455a-9c76-9735991c0cb2 2bd15992-4d28-455a-9c76-9735991c0cb2}" --- PASS: TestRaft_Voting (0.31s) === RUN TestRaft_ProtocolVersion_RejectRPC testLoggerAdapter verbose: server-af220f1d-c3df-5c25-537f-4d761cc41a81: 2021-08-25T09:28:16.182Z [INFO] server-af220f1d-c3df-5c25-537f-4d761cc41a81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af220f1d-c3df-5c25-537f-4d761cc41a81 Address:af220f1d-c3df-5c25-537f-4d761cc41a81} {Suffrage:Voter ID:server-04fc31fd-c621-9a31-78d8-e89c80f91b5d Address:04fc31fd-c621-9a31-78d8-e89c80f91b5d} {Suffrage:Voter ID:server-5f25179c-a632-ca43-c5df-f03743459518 Address:5f25179c-a632-ca43-c5df-f03743459518}]" testing.go:149: server-af220f1d-c3df-5c25-537f-4d761cc41a81: 2021-08-25T09:28:16.182Z [INFO] server-af220f1d-c3df-5c25-537f-4d761cc41a81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af220f1d-c3df-5c25-537f-4d761cc41a81 Address:af220f1d-c3df-5c25-537f-4d761cc41a81} {Suffrage:Voter ID:server-04fc31fd-c621-9a31-78d8-e89c80f91b5d Address:04fc31fd-c621-9a31-78d8-e89c80f91b5d} {Suffrage:Voter ID:server-5f25179c-a632-ca43-c5df-f03743459518 Address:5f25179c-a632-ca43-c5df-f03743459518}]" testLoggerAdapter verbose: server-af220f1d-c3df-5c25-537f-4d761cc41a81: 2021-08-25T09:28:16.182Z [INFO] server-af220f1d-c3df-5c25-537f-4d761cc41a81: entering follower state: follower="Node at af220f1d-c3df-5c25-537f-4d761cc41a81 [Follower]" leader= testing.go:149: server-af220f1d-c3df-5c25-537f-4d761cc41a81: 2021-08-25T09:28:16.182Z [INFO] server-af220f1d-c3df-5c25-537f-4d761cc41a81: entering follower state: follower="Node at af220f1d-c3df-5c25-537f-4d761cc41a81 [Follower]" leader= testLoggerAdapter verbose: server-04fc31fd-c621-9a31-78d8-e89c80f91b5d: 2021-08-25T09:28:16.182Z [INFO] server-04fc31fd-c621-9a31-78d8-e89c80f91b5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af220f1d-c3df-5c25-537f-4d761cc41a81 Address:af220f1d-c3df-5c25-537f-4d761cc41a81} {Suffrage:Voter ID:server-04fc31fd-c621-9a31-78d8-e89c80f91b5d Address:04fc31fd-c621-9a31-78d8-e89c80f91b5d} {Suffrage:Voter ID:server-5f25179c-a632-ca43-c5df-f03743459518 Address:5f25179c-a632-ca43-c5df-f03743459518}]" testing.go:149: server-04fc31fd-c621-9a31-78d8-e89c80f91b5d: 2021-08-25T09:28:16.182Z [INFO] server-04fc31fd-c621-9a31-78d8-e89c80f91b5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af220f1d-c3df-5c25-537f-4d761cc41a81 Address:af220f1d-c3df-5c25-537f-4d761cc41a81} {Suffrage:Voter ID:server-04fc31fd-c621-9a31-78d8-e89c80f91b5d Address:04fc31fd-c621-9a31-78d8-e89c80f91b5d} {Suffrage:Voter ID:server-5f25179c-a632-ca43-c5df-f03743459518 Address:5f25179c-a632-ca43-c5df-f03743459518}]" testLoggerAdapter verbose: server-04fc31fd-c621-9a31-78d8-e89c80f91b5d: 2021-08-25T09:28:16.182Z [INFO] server-04fc31fd-c621-9a31-78d8-e89c80f91b5d: entering follower state: follower="Node at 04fc31fd-c621-9a31-78d8-e89c80f91b5d [Follower]" leader= testing.go:149: server-04fc31fd-c621-9a31-78d8-e89c80f91b5d: 2021-08-25T09:28:16.182Z [INFO] server-04fc31fd-c621-9a31-78d8-e89c80f91b5d: entering follower state: follower="Node at 04fc31fd-c621-9a31-78d8-e89c80f91b5d [Follower]" leader= testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.183Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af220f1d-c3df-5c25-537f-4d761cc41a81 Address:af220f1d-c3df-5c25-537f-4d761cc41a81} {Suffrage:Voter ID:server-04fc31fd-c621-9a31-78d8-e89c80f91b5d Address:04fc31fd-c621-9a31-78d8-e89c80f91b5d} {Suffrage:Voter ID:server-5f25179c-a632-ca43-c5df-f03743459518 Address:5f25179c-a632-ca43-c5df-f03743459518}]" testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.183Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af220f1d-c3df-5c25-537f-4d761cc41a81 Address:af220f1d-c3df-5c25-537f-4d761cc41a81} {Suffrage:Voter ID:server-04fc31fd-c621-9a31-78d8-e89c80f91b5d Address:04fc31fd-c621-9a31-78d8-e89c80f91b5d} {Suffrage:Voter ID:server-5f25179c-a632-ca43-c5df-f03743459518 Address:5f25179c-a632-ca43-c5df-f03743459518}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.183Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:16.183Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.183Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: entering follower state: follower="Node at 5f25179c-a632-ca43-c5df-f03743459518 [Follower]" leader= testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.183Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: entering follower state: follower="Node at 5f25179c-a632-ca43-c5df-f03743459518 [Follower]" leader= testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.258Z [WARN] server-5f25179c-a632-ca43-c5df-f03743459518: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.258Z [WARN] server-5f25179c-a632-ca43-c5df-f03743459518: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: entering candidate state: node="Node at 5f25179c-a632-ca43-c5df-f03743459518 [Candidate]" term=2 testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: entering candidate state: node="Node at 5f25179c-a632-ca43-c5df-f03743459518 [Candidate]" term=2 testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: election won: tally=2 testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: election won: tally=2 testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: entering leader state: leader="Node at 5f25179c-a632-ca43-c5df-f03743459518 [Leader]" testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: entering leader state: leader="Node at 5f25179c-a632-ca43-c5df-f03743459518 [Leader]" testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: added peer, starting replication: peer=server-af220f1d-c3df-5c25-537f-4d761cc41a81 testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: added peer, starting replication: peer=server-af220f1d-c3df-5c25-537f-4d761cc41a81 testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: added peer, starting replication: peer=server-04fc31fd-c621-9a31-78d8-e89c80f91b5d testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: added peer, starting replication: peer=server-04fc31fd-c621-9a31-78d8-e89c80f91b5d testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: pipelining replication: peer="{Voter server-af220f1d-c3df-5c25-537f-4d761cc41a81 af220f1d-c3df-5c25-537f-4d761cc41a81}" testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: pipelining replication: peer="{Voter server-af220f1d-c3df-5c25-537f-4d761cc41a81 af220f1d-c3df-5c25-537f-4d761cc41a81}" testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: pipelining replication: peer="{Voter server-04fc31fd-c621-9a31-78d8-e89c80f91b5d 04fc31fd-c621-9a31-78d8-e89c80f91b5d}" testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: pipelining replication: peer="{Voter server-04fc31fd-c621-9a31-78d8-e89c80f91b5d 04fc31fd-c621-9a31-78d8-e89c80f91b5d}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.364Z [INFO] stable state for Follower reached at 2021-08-25 09:28:16.259358737 +0000 UTC m=+16.023574646 (2 nodes), 75.96338ms from start of poll, 77.325174ms from cluster start. Timeout at 2021-08-25 09:28:16.364494885 +0000 UTC m=+16.128710854, 105.136208ms after stability testing.go:149: cluster: 2021-08-25T09:28:16.364Z [INFO] stable state for Follower reached at 2021-08-25 09:28:16.259358737 +0000 UTC m=+16.023574646 (2 nodes), 75.96338ms from start of poll, 77.325174ms from cluster start. Timeout at 2021-08-25 09:28:16.364494885 +0000 UTC m=+16.128710854, 105.136208ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.364Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:16.364Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.469Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.364816216 +0000 UTC m=+16.129032125 (1 nodes), 2.42µs from start of poll, 182.782653ms from cluster start. Timeout at 2021-08-25 09:28:16.46988655 +0000 UTC m=+16.234102499, 105.070374ms after stability testing.go:149: cluster: 2021-08-25T09:28:16.469Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.364816216 +0000 UTC m=+16.129032125 (1 nodes), 2.42µs from start of poll, 182.782653ms from cluster start. Timeout at 2021-08-25 09:28:16.46988655 +0000 UTC m=+16.234102499, 105.070374ms after stability testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.470Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: aborting pipeline replication: peer="{Voter server-04fc31fd-c621-9a31-78d8-e89c80f91b5d 04fc31fd-c621-9a31-78d8-e89c80f91b5d}" testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.470Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: aborting pipeline replication: peer="{Voter server-04fc31fd-c621-9a31-78d8-e89c80f91b5d 04fc31fd-c621-9a31-78d8-e89c80f91b5d}" testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.470Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: aborting pipeline replication: peer="{Voter server-af220f1d-c3df-5c25-537f-4d761cc41a81 af220f1d-c3df-5c25-537f-4d761cc41a81}" testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.470Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: aborting pipeline replication: peer="{Voter server-af220f1d-c3df-5c25-537f-4d761cc41a81 af220f1d-c3df-5c25-537f-4d761cc41a81}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.29s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.474Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: initial configuration: index=1 servers="[{Suffrage:Voter ID:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 Address:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8} {Suffrage:Voter ID:d773f2e2-2931-7d53-9203-437d0936b98f Address:d773f2e2-2931-7d53-9203-437d0936b98f}]" testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.474Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: initial configuration: index=1 servers="[{Suffrage:Voter ID:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 Address:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8} {Suffrage:Voter ID:d773f2e2-2931-7d53-9203-437d0936b98f Address:d773f2e2-2931-7d53-9203-437d0936b98f}]" testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.474Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: entering follower state: follower="Node at 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 [Follower]" leader= testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.474Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: entering follower state: follower="Node at 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 [Follower]" leader= testLoggerAdapter verbose: d773f2e2-2931-7d53-9203-437d0936b98f: 2021-08-25T09:28:16.474Z [INFO] d773f2e2-2931-7d53-9203-437d0936b98f: initial configuration: index=1 servers="[{Suffrage:Voter ID:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 Address:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8} {Suffrage:Voter ID:d773f2e2-2931-7d53-9203-437d0936b98f Address:d773f2e2-2931-7d53-9203-437d0936b98f}]" testing.go:149: d773f2e2-2931-7d53-9203-437d0936b98f: 2021-08-25T09:28:16.474Z [INFO] d773f2e2-2931-7d53-9203-437d0936b98f: initial configuration: index=1 servers="[{Suffrage:Voter ID:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 Address:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8} {Suffrage:Voter ID:d773f2e2-2931-7d53-9203-437d0936b98f Address:d773f2e2-2931-7d53-9203-437d0936b98f}]" testLoggerAdapter verbose: d773f2e2-2931-7d53-9203-437d0936b98f: 2021-08-25T09:28:16.474Z [INFO] d773f2e2-2931-7d53-9203-437d0936b98f: entering follower state: follower="Node at d773f2e2-2931-7d53-9203-437d0936b98f [Follower]" leader= testing.go:149: d773f2e2-2931-7d53-9203-437d0936b98f: 2021-08-25T09:28:16.474Z [INFO] d773f2e2-2931-7d53-9203-437d0936b98f: entering follower state: follower="Node at d773f2e2-2931-7d53-9203-437d0936b98f [Follower]" leader= testLoggerAdapter verbose: e526e47f-5207-5caa-3eae-ebfd83ed2534: 2021-08-25T09:28:16.475Z [INFO] e526e47f-5207-5caa-3eae-ebfd83ed2534: initial configuration: index=0 servers=[] testing.go:149: e526e47f-5207-5caa-3eae-ebfd83ed2534: 2021-08-25T09:28:16.475Z [INFO] e526e47f-5207-5caa-3eae-ebfd83ed2534: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.475Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:16.475Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: e526e47f-5207-5caa-3eae-ebfd83ed2534: 2021-08-25T09:28:16.475Z [INFO] e526e47f-5207-5caa-3eae-ebfd83ed2534: entering follower state: follower="Node at e526e47f-5207-5caa-3eae-ebfd83ed2534 [Follower]" leader= testing.go:149: e526e47f-5207-5caa-3eae-ebfd83ed2534: 2021-08-25T09:28:16.475Z [INFO] e526e47f-5207-5caa-3eae-ebfd83ed2534: entering follower state: follower="Node at e526e47f-5207-5caa-3eae-ebfd83ed2534 [Follower]" leader= testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.526Z [WARN] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: heartbeat timeout reached, starting election: last-leader= testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.526Z [WARN] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.526Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: entering candidate state: node="Node at 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 [Candidate]" term=2 testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.526Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: entering candidate state: node="Node at 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 [Candidate]" term=2 testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.527Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: election won: tally=2 testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.527Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: election won: tally=2 testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.527Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: entering leader state: leader="Node at 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 [Leader]" testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.527Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: entering leader state: leader="Node at 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 [Leader]" testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.527Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: added peer, starting replication: peer=d773f2e2-2931-7d53-9203-437d0936b98f testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.527Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: added peer, starting replication: peer=d773f2e2-2931-7d53-9203-437d0936b98f testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.527Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: pipelining replication: peer="{Voter d773f2e2-2931-7d53-9203-437d0936b98f d773f2e2-2931-7d53-9203-437d0936b98f}" testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.527Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: pipelining replication: peer="{Voter d773f2e2-2931-7d53-9203-437d0936b98f d773f2e2-2931-7d53-9203-437d0936b98f}" testLoggerAdapter verbose: e526e47f-5207-5caa-3eae-ebfd83ed2534: 2021-08-25T09:28:16.545Z [WARN] e526e47f-5207-5caa-3eae-ebfd83ed2534: no known peers, aborting election testing.go:149: e526e47f-5207-5caa-3eae-ebfd83ed2534: 2021-08-25T09:28:16.545Z [WARN] e526e47f-5207-5caa-3eae-ebfd83ed2534: no known peers, aborting election testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.632Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.527397585 +0000 UTC m=+16.291613494 (1 nodes), 51.930953ms from start of poll, 53.57578ms from cluster start. Timeout at 2021-08-25 09:28:16.632482499 +0000 UTC m=+16.396698468, 105.084974ms after stability testing.go:149: cluster: 2021-08-25T09:28:16.632Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.527397585 +0000 UTC m=+16.291613494 (1 nodes), 51.930953ms from start of poll, 53.57578ms from cluster start. Timeout at 2021-08-25 09:28:16.632482499 +0000 UTC m=+16.396698468, 105.084974ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.632Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:16.632Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.737Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.632755234 +0000 UTC m=+16.396971123 (1 nodes), 2.7µs from start of poll, 158.933409ms from cluster start. Timeout at 2021-08-25 09:28:16.737886703 +0000 UTC m=+16.502102692, 105.131569ms after stability testing.go:149: cluster: 2021-08-25T09:28:16.737Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.632755234 +0000 UTC m=+16.396971123 (1 nodes), 2.7µs from start of poll, 158.933409ms from cluster start. Timeout at 2021-08-25 09:28:16.737886703 +0000 UTC m=+16.502102692, 105.131569ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.738Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:16.738Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.843Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.738193854 +0000 UTC m=+16.502409763 (1 nodes), 2.18µs from start of poll, 264.372049ms from cluster start. Timeout at 2021-08-25 09:28:16.843338782 +0000 UTC m=+16.607554771, 105.145008ms after stability testing.go:149: cluster: 2021-08-25T09:28:16.843Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.738193854 +0000 UTC m=+16.502409763 (1 nodes), 2.18µs from start of poll, 264.372049ms from cluster start. Timeout at 2021-08-25 09:28:16.843338782 +0000 UTC m=+16.607554771, 105.145008ms after stability testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.843Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: updating configuration: command=AddStaging server-id=e526e47f-5207-5caa-3eae-ebfd83ed2534 server-addr=e526e47f-5207-5caa-3eae-ebfd83ed2534 servers="[{Suffrage:Voter ID:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 Address:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8} {Suffrage:Voter ID:d773f2e2-2931-7d53-9203-437d0936b98f Address:d773f2e2-2931-7d53-9203-437d0936b98f} {Suffrage:Voter ID:e526e47f-5207-5caa-3eae-ebfd83ed2534 Address:e526e47f-5207-5caa-3eae-ebfd83ed2534}]" testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.843Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: updating configuration: command=AddStaging server-id=e526e47f-5207-5caa-3eae-ebfd83ed2534 server-addr=e526e47f-5207-5caa-3eae-ebfd83ed2534 servers="[{Suffrage:Voter ID:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 Address:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8} {Suffrage:Voter ID:d773f2e2-2931-7d53-9203-437d0936b98f Address:d773f2e2-2931-7d53-9203-437d0936b98f} {Suffrage:Voter ID:e526e47f-5207-5caa-3eae-ebfd83ed2534 Address:e526e47f-5207-5caa-3eae-ebfd83ed2534}]" testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.843Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: added peer, starting replication: peer=e526e47f-5207-5caa-3eae-ebfd83ed2534 testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.843Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: added peer, starting replication: peer=e526e47f-5207-5caa-3eae-ebfd83ed2534 testLoggerAdapter verbose: e526e47f-5207-5caa-3eae-ebfd83ed2534: 2021-08-25T09:28:16.844Z [WARN] e526e47f-5207-5caa-3eae-ebfd83ed2534: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: e526e47f-5207-5caa-3eae-ebfd83ed2534: 2021-08-25T09:28:16.844Z [WARN] e526e47f-5207-5caa-3eae-ebfd83ed2534: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.844Z [WARN] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: appendEntries rejected, sending older logs: peer="{Voter e526e47f-5207-5caa-3eae-ebfd83ed2534 e526e47f-5207-5caa-3eae-ebfd83ed2534}" next=1 testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.844Z [WARN] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: appendEntries rejected, sending older logs: peer="{Voter e526e47f-5207-5caa-3eae-ebfd83ed2534 e526e47f-5207-5caa-3eae-ebfd83ed2534}" next=1 testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.844Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: pipelining replication: peer="{Voter e526e47f-5207-5caa-3eae-ebfd83ed2534 e526e47f-5207-5caa-3eae-ebfd83ed2534}" testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.844Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: pipelining replication: peer="{Voter e526e47f-5207-5caa-3eae-ebfd83ed2534 e526e47f-5207-5caa-3eae-ebfd83ed2534}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.849Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:16.849Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.954Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.849483091 +0000 UTC m=+16.613698980 (1 nodes), 2.08µs from start of poll, 375.661266ms from cluster start. Timeout at 2021-08-25 09:28:16.95462184 +0000 UTC m=+16.718837809, 105.138829ms after stability testing.go:149: cluster: 2021-08-25T09:28:16.954Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.849483091 +0000 UTC m=+16.613698980 (1 nodes), 2.08µs from start of poll, 375.661266ms from cluster start. Timeout at 2021-08-25 09:28:16.95462184 +0000 UTC m=+16.718837809, 105.138829ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.954Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:16.954Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.060Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.954916492 +0000 UTC m=+16.719132381 (1 nodes), 2.24µs from start of poll, 481.094667ms from cluster start. Timeout at 2021-08-25 09:28:17.059997086 +0000 UTC m=+16.824213035, 105.080654ms after stability testing.go:149: cluster: 2021-08-25T09:28:17.060Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.954916492 +0000 UTC m=+16.719132381 (1 nodes), 2.24µs from start of poll, 481.094667ms from cluster start. Timeout at 2021-08-25 09:28:17.059997086 +0000 UTC m=+16.824213035, 105.080654ms after stability testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:17.060Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: updating configuration: command=RemoveServer server-id=e526e47f-5207-5caa-3eae-ebfd83ed2534 server-addr= servers="[{Suffrage:Voter ID:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 Address:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8} {Suffrage:Voter ID:d773f2e2-2931-7d53-9203-437d0936b98f Address:d773f2e2-2931-7d53-9203-437d0936b98f}]" testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:17.060Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: updating configuration: command=RemoveServer server-id=e526e47f-5207-5caa-3eae-ebfd83ed2534 server-addr= servers="[{Suffrage:Voter ID:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 Address:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8} {Suffrage:Voter ID:d773f2e2-2931-7d53-9203-437d0936b98f Address:d773f2e2-2931-7d53-9203-437d0936b98f}]" testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:17.060Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: removed peer, stopping replication: peer=e526e47f-5207-5caa-3eae-ebfd83ed2534 last-index=4 testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:17.060Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: removed peer, stopping replication: peer=e526e47f-5207-5caa-3eae-ebfd83ed2534 last-index=4 testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:17.060Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: aborting pipeline replication: peer="{Voter e526e47f-5207-5caa-3eae-ebfd83ed2534 e526e47f-5207-5caa-3eae-ebfd83ed2534}" testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:17.060Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: aborting pipeline replication: peer="{Voter e526e47f-5207-5caa-3eae-ebfd83ed2534 e526e47f-5207-5caa-3eae-ebfd83ed2534}" testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:17.060Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: aborting pipeline replication: peer="{Voter d773f2e2-2931-7d53-9203-437d0936b98f d773f2e2-2931-7d53-9203-437d0936b98f}" testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:17.060Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: aborting pipeline replication: peer="{Voter d773f2e2-2931-7d53-9203-437d0936b98f d773f2e2-2931-7d53-9203-437d0936b98f}" --- PASS: TestRaft_ProtocolVersion_Upgrade_1_2 (0.59s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 testLoggerAdapter verbose: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.063Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: initial configuration: index=1 servers="[{Suffrage:Voter ID:18323061-8241-df37-0c12-8c3d6b8b309c Address:18323061-8241-df37-0c12-8c3d6b8b309c} {Suffrage:Voter ID:07270d5a-1384-f258-112b-3142e87f0e0b Address:07270d5a-1384-f258-112b-3142e87f0e0b}]" testing.go:149: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.063Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: initial configuration: index=1 servers="[{Suffrage:Voter ID:18323061-8241-df37-0c12-8c3d6b8b309c Address:18323061-8241-df37-0c12-8c3d6b8b309c} {Suffrage:Voter ID:07270d5a-1384-f258-112b-3142e87f0e0b Address:07270d5a-1384-f258-112b-3142e87f0e0b}]" testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.064Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: initial configuration: index=1 servers="[{Suffrage:Voter ID:18323061-8241-df37-0c12-8c3d6b8b309c Address:18323061-8241-df37-0c12-8c3d6b8b309c} {Suffrage:Voter ID:07270d5a-1384-f258-112b-3142e87f0e0b Address:07270d5a-1384-f258-112b-3142e87f0e0b}]" testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.064Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: initial configuration: index=1 servers="[{Suffrage:Voter ID:18323061-8241-df37-0c12-8c3d6b8b309c Address:18323061-8241-df37-0c12-8c3d6b8b309c} {Suffrage:Voter ID:07270d5a-1384-f258-112b-3142e87f0e0b Address:07270d5a-1384-f258-112b-3142e87f0e0b}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.064Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:17.064Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.064Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: entering follower state: follower="Node at 18323061-8241-df37-0c12-8c3d6b8b309c [Follower]" leader= testing.go:149: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.064Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: entering follower state: follower="Node at 18323061-8241-df37-0c12-8c3d6b8b309c [Follower]" leader= testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.064Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: entering follower state: follower="Node at 07270d5a-1384-f258-112b-3142e87f0e0b [Follower]" leader= testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.064Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: entering follower state: follower="Node at 07270d5a-1384-f258-112b-3142e87f0e0b [Follower]" leader= testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.144Z [WARN] 07270d5a-1384-f258-112b-3142e87f0e0b: heartbeat timeout reached, starting election: last-leader= testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.144Z [WARN] 07270d5a-1384-f258-112b-3142e87f0e0b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.144Z [WARN] 18323061-8241-df37-0c12-8c3d6b8b309c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.144Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: entering candidate state: node="Node at 07270d5a-1384-f258-112b-3142e87f0e0b [Candidate]" term=2 testing.go:149: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.144Z [WARN] 18323061-8241-df37-0c12-8c3d6b8b309c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.145Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: entering candidate state: node="Node at 18323061-8241-df37-0c12-8c3d6b8b309c [Candidate]" term=2 testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.144Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: entering candidate state: node="Node at 07270d5a-1384-f258-112b-3142e87f0e0b [Candidate]" term=2 testing.go:149: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.145Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: entering candidate state: node="Node at 18323061-8241-df37-0c12-8c3d6b8b309c [Candidate]" term=2 testLoggerAdapter verbose: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.145Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: duplicate requestVote for same term: term=2 testing.go:149: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.145Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.145Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: duplicate requestVote for same term: term=2 testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.145Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.209Z [WARN] 07270d5a-1384-f258-112b-3142e87f0e0b: Election timeout reached, restarting election testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.209Z [WARN] 07270d5a-1384-f258-112b-3142e87f0e0b: Election timeout reached, restarting election testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: entering candidate state: node="Node at 07270d5a-1384-f258-112b-3142e87f0e0b [Candidate]" term=3 testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: entering candidate state: node="Node at 07270d5a-1384-f258-112b-3142e87f0e0b [Candidate]" term=3 testLoggerAdapter verbose: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.210Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: entering follower state: follower="Node at 18323061-8241-df37-0c12-8c3d6b8b309c [Follower]" leader= testing.go:149: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.210Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: entering follower state: follower="Node at 18323061-8241-df37-0c12-8c3d6b8b309c [Follower]" leader= testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: election won: tally=2 testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: election won: tally=2 testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: entering leader state: leader="Node at 07270d5a-1384-f258-112b-3142e87f0e0b [Leader]" testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: entering leader state: leader="Node at 07270d5a-1384-f258-112b-3142e87f0e0b [Leader]" testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: added peer, starting replication: peer=18323061-8241-df37-0c12-8c3d6b8b309c testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: added peer, starting replication: peer=18323061-8241-df37-0c12-8c3d6b8b309c testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: pipelining replication: peer="{Voter 18323061-8241-df37-0c12-8c3d6b8b309c 18323061-8241-df37-0c12-8c3d6b8b309c}" testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: pipelining replication: peer="{Voter 18323061-8241-df37-0c12-8c3d6b8b309c 18323061-8241-df37-0c12-8c3d6b8b309c}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.316Z [INFO] stable state for Follower reached at 2021-08-25 09:28:17.210901721 +0000 UTC m=+16.975117610 (1 nodes), 146.534701ms from start of poll, 147.1927ms from cluster start. Timeout at 2021-08-25 09:28:17.316014672 +0000 UTC m=+17.080230721, 105.113111ms after stability testing.go:149: cluster: 2021-08-25T09:28:17.316Z [INFO] stable state for Follower reached at 2021-08-25 09:28:17.210901721 +0000 UTC m=+16.975117610 (1 nodes), 146.534701ms from start of poll, 147.1927ms from cluster start. Timeout at 2021-08-25 09:28:17.316014672 +0000 UTC m=+17.080230721, 105.113111ms after stability testLoggerAdapter verbose: server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: 2021-08-25T09:28:17.317Z [INFO] server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: initial configuration: index=0 servers=[] testing.go:149: server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: 2021-08-25T09:28:17.317Z [INFO] server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.317Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:17.317Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: 2021-08-25T09:28:17.317Z [INFO] server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: entering follower state: follower="Node at e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 [Follower]" leader= testing.go:149: server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: 2021-08-25T09:28:17.317Z [INFO] server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: entering follower state: follower="Node at e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 [Follower]" leader= testLoggerAdapter verbose: server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: 2021-08-25T09:28:17.415Z [WARN] server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: no known peers, aborting election testing.go:149: server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: 2021-08-25T09:28:17.415Z [WARN] server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: no known peers, aborting election testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.422Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.317206621 +0000 UTC m=+17.081422510 (1 nodes), 1.959µs from start of poll, 253.4976ms from cluster start. Timeout at 2021-08-25 09:28:17.422336791 +0000 UTC m=+17.186552700, 105.13019ms after stability testing.go:149: cluster: 2021-08-25T09:28:17.422Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.317206621 +0000 UTC m=+17.081422510 (1 nodes), 1.959µs from start of poll, 253.4976ms from cluster start. Timeout at 2021-08-25 09:28:17.422336791 +0000 UTC m=+17.186552700, 105.13019ms after stability testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.422Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: updating configuration: command=AddStaging server-id=server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 server-addr=e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 servers="[{Suffrage:Voter ID:18323061-8241-df37-0c12-8c3d6b8b309c Address:18323061-8241-df37-0c12-8c3d6b8b309c} {Suffrage:Voter ID:07270d5a-1384-f258-112b-3142e87f0e0b Address:07270d5a-1384-f258-112b-3142e87f0e0b} {Suffrage:Voter ID:server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 Address:e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}]" testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.422Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: updating configuration: command=AddStaging server-id=server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 server-addr=e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 servers="[{Suffrage:Voter ID:18323061-8241-df37-0c12-8c3d6b8b309c Address:18323061-8241-df37-0c12-8c3d6b8b309c} {Suffrage:Voter ID:07270d5a-1384-f258-112b-3142e87f0e0b Address:07270d5a-1384-f258-112b-3142e87f0e0b} {Suffrage:Voter ID:server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 Address:e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}]" testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.422Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: added peer, starting replication: peer=server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.422Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: added peer, starting replication: peer=server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 testLoggerAdapter verbose: server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: 2021-08-25T09:28:17.422Z [WARN] server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: 2021-08-25T09:28:17.422Z [WARN] server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.423Z [WARN] 07270d5a-1384-f258-112b-3142e87f0e0b: appendEntries rejected, sending older logs: peer="{Voter server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}" next=1 testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.423Z [WARN] 07270d5a-1384-f258-112b-3142e87f0e0b: appendEntries rejected, sending older logs: peer="{Voter server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}" next=1 testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.423Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: pipelining replication: peer="{Voter server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}" testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.423Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: pipelining replication: peer="{Voter server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.428Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:17.428Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.533Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.428436224 +0000 UTC m=+17.192652113 (1 nodes), 2.28µs from start of poll, 364.727203ms from cluster start. Timeout at 2021-08-25 09:28:17.533530896 +0000 UTC m=+17.297746805, 105.094692ms after stability testing.go:149: cluster: 2021-08-25T09:28:17.533Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.428436224 +0000 UTC m=+17.192652113 (1 nodes), 2.28µs from start of poll, 364.727203ms from cluster start. Timeout at 2021-08-25 09:28:17.533530896 +0000 UTC m=+17.297746805, 105.094692ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.533Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:17.533Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.638Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.533790332 +0000 UTC m=+17.298006221 (1 nodes), 2.139µs from start of poll, 470.081311ms from cluster start. Timeout at 2021-08-25 09:28:17.638846029 +0000 UTC m=+17.403061898, 105.055677ms after stability testing.go:149: cluster: 2021-08-25T09:28:17.638Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.533790332 +0000 UTC m=+17.298006221 (1 nodes), 2.139µs from start of poll, 470.081311ms from cluster start. Timeout at 2021-08-25 09:28:17.638846029 +0000 UTC m=+17.403061898, 105.055677ms after stability testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.639Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: updating configuration: command=RemoveServer server-id=18323061-8241-df37-0c12-8c3d6b8b309c server-addr= servers="[{Suffrage:Voter ID:07270d5a-1384-f258-112b-3142e87f0e0b Address:07270d5a-1384-f258-112b-3142e87f0e0b} {Suffrage:Voter ID:server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 Address:e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}]" testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.639Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: updating configuration: command=RemoveServer server-id=18323061-8241-df37-0c12-8c3d6b8b309c server-addr= servers="[{Suffrage:Voter ID:07270d5a-1384-f258-112b-3142e87f0e0b Address:07270d5a-1384-f258-112b-3142e87f0e0b} {Suffrage:Voter ID:server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 Address:e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}]" testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.639Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: removed peer, stopping replication: peer=18323061-8241-df37-0c12-8c3d6b8b309c last-index=4 testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.639Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: removed peer, stopping replication: peer=18323061-8241-df37-0c12-8c3d6b8b309c last-index=4 testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.639Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: aborting pipeline replication: peer="{Voter 18323061-8241-df37-0c12-8c3d6b8b309c 18323061-8241-df37-0c12-8c3d6b8b309c}" testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.639Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: aborting pipeline replication: peer="{Voter 18323061-8241-df37-0c12-8c3d6b8b309c 18323061-8241-df37-0c12-8c3d6b8b309c}" testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.639Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: aborting pipeline replication: peer="{Voter server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}" testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.639Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: aborting pipeline replication: peer="{Voter server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}" --- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.58s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.643Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91e07238-9313-b639-00d7-4caf2cd4ff45 Address:91e07238-9313-b639-00d7-4caf2cd4ff45} {Suffrage:Voter ID:server-1d41a700-ab7c-4948-cf83-b63a833e14fc Address:1d41a700-ab7c-4948-cf83-b63a833e14fc} {Suffrage:Voter ID:server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 Address:7b9835ce-b8b3-5c56-a8f5-df64f9056b76}]" testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.643Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91e07238-9313-b639-00d7-4caf2cd4ff45 Address:91e07238-9313-b639-00d7-4caf2cd4ff45} {Suffrage:Voter ID:server-1d41a700-ab7c-4948-cf83-b63a833e14fc Address:1d41a700-ab7c-4948-cf83-b63a833e14fc} {Suffrage:Voter ID:server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 Address:7b9835ce-b8b3-5c56-a8f5-df64f9056b76}]" testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.643Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: entering follower state: follower="Node at 91e07238-9313-b639-00d7-4caf2cd4ff45 [Follower]" leader= testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.643Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: entering follower state: follower="Node at 91e07238-9313-b639-00d7-4caf2cd4ff45 [Follower]" leader= testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.643Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91e07238-9313-b639-00d7-4caf2cd4ff45 Address:91e07238-9313-b639-00d7-4caf2cd4ff45} {Suffrage:Voter ID:server-1d41a700-ab7c-4948-cf83-b63a833e14fc Address:1d41a700-ab7c-4948-cf83-b63a833e14fc} {Suffrage:Voter ID:server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 Address:7b9835ce-b8b3-5c56-a8f5-df64f9056b76}]" testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.643Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91e07238-9313-b639-00d7-4caf2cd4ff45 Address:91e07238-9313-b639-00d7-4caf2cd4ff45} {Suffrage:Voter ID:server-1d41a700-ab7c-4948-cf83-b63a833e14fc Address:1d41a700-ab7c-4948-cf83-b63a833e14fc} {Suffrage:Voter ID:server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 Address:7b9835ce-b8b3-5c56-a8f5-df64f9056b76}]" testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.643Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: entering follower state: follower="Node at 1d41a700-ab7c-4948-cf83-b63a833e14fc [Follower]" leader= testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.643Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: entering follower state: follower="Node at 1d41a700-ab7c-4948-cf83-b63a833e14fc [Follower]" leader= testLoggerAdapter verbose: server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76: 2021-08-25T09:28:17.643Z [INFO] server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91e07238-9313-b639-00d7-4caf2cd4ff45 Address:91e07238-9313-b639-00d7-4caf2cd4ff45} {Suffrage:Voter ID:server-1d41a700-ab7c-4948-cf83-b63a833e14fc Address:1d41a700-ab7c-4948-cf83-b63a833e14fc} {Suffrage:Voter ID:server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 Address:7b9835ce-b8b3-5c56-a8f5-df64f9056b76}]" testing.go:149: server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76: 2021-08-25T09:28:17.643Z [INFO] server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91e07238-9313-b639-00d7-4caf2cd4ff45 Address:91e07238-9313-b639-00d7-4caf2cd4ff45} {Suffrage:Voter ID:server-1d41a700-ab7c-4948-cf83-b63a833e14fc Address:1d41a700-ab7c-4948-cf83-b63a833e14fc} {Suffrage:Voter ID:server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 Address:7b9835ce-b8b3-5c56-a8f5-df64f9056b76}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.643Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:17.643Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76: 2021-08-25T09:28:17.643Z [INFO] server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76: entering follower state: follower="Node at 7b9835ce-b8b3-5c56-a8f5-df64f9056b76 [Follower]" leader= testing.go:149: server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76: 2021-08-25T09:28:17.643Z [INFO] server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76: entering follower state: follower="Node at 7b9835ce-b8b3-5c56-a8f5-df64f9056b76 [Follower]" leader= testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.716Z [WARN] server-91e07238-9313-b639-00d7-4caf2cd4ff45: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.716Z [WARN] server-91e07238-9313-b639-00d7-4caf2cd4ff45: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.716Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: entering candidate state: node="Node at 91e07238-9313-b639-00d7-4caf2cd4ff45 [Candidate]" term=2 testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.716Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: entering candidate state: node="Node at 91e07238-9313-b639-00d7-4caf2cd4ff45 [Candidate]" term=2 testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.716Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: election won: tally=2 testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.716Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: election won: tally=2 testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.716Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: entering leader state: leader="Node at 91e07238-9313-b639-00d7-4caf2cd4ff45 [Leader]" testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.716Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: entering leader state: leader="Node at 91e07238-9313-b639-00d7-4caf2cd4ff45 [Leader]" testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.717Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: added peer, starting replication: peer=server-1d41a700-ab7c-4948-cf83-b63a833e14fc testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.717Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: added peer, starting replication: peer=server-1d41a700-ab7c-4948-cf83-b63a833e14fc testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.717Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: added peer, starting replication: peer=server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.717Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: added peer, starting replication: peer=server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.717Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: pipelining replication: peer="{Voter server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 7b9835ce-b8b3-5c56-a8f5-df64f9056b76}" testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.717Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: pipelining replication: peer="{Voter server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 7b9835ce-b8b3-5c56-a8f5-df64f9056b76}" testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.717Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: pipelining replication: peer="{Voter server-1d41a700-ab7c-4948-cf83-b63a833e14fc 1d41a700-ab7c-4948-cf83-b63a833e14fc}" testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.717Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: pipelining replication: peer="{Voter server-1d41a700-ab7c-4948-cf83-b63a833e14fc 1d41a700-ab7c-4948-cf83-b63a833e14fc}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.822Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.717666283 +0000 UTC m=+17.481882172 (1 nodes), 73.747426ms from start of poll, 74.761892ms from cluster start. Timeout at 2021-08-25 09:28:17.822807931 +0000 UTC m=+17.587023880, 105.141708ms after stability testing.go:149: cluster: 2021-08-25T09:28:17.822Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.717666283 +0000 UTC m=+17.481882172 (1 nodes), 73.747426ms from start of poll, 74.761892ms from cluster start. Timeout at 2021-08-25 09:28:17.822807931 +0000 UTC m=+17.587023880, 105.141708ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.823Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:17.823Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.928Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.823074347 +0000 UTC m=+17.587290236 (1 nodes), 2.18µs from start of poll, 180.169956ms from cluster start. Timeout at 2021-08-25 09:28:17.928154841 +0000 UTC m=+17.692370789, 105.080553ms after stability testing.go:149: cluster: 2021-08-25T09:28:17.928Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.823074347 +0000 UTC m=+17.587290236 (1 nodes), 2.18µs from start of poll, 180.169956ms from cluster start. Timeout at 2021-08-25 09:28:17.928154841 +0000 UTC m=+17.692370789, 105.080553ms after stability testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: entering candidate state: node="Node at 1d41a700-ab7c-4948-cf83-b63a833e14fc [Candidate]" term=3 testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: entering candidate state: node="Node at 1d41a700-ab7c-4948-cf83-b63a833e14fc [Candidate]" term=3 testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: election won: tally=2 testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: election won: tally=2 testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: entering leader state: leader="Node at 1d41a700-ab7c-4948-cf83-b63a833e14fc [Leader]" testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.928Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: entering follower state: follower="Node at 91e07238-9313-b639-00d7-4caf2cd4ff45 [Follower]" leader= testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: entering leader state: leader="Node at 1d41a700-ab7c-4948-cf83-b63a833e14fc [Leader]" testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.928Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: entering follower state: follower="Node at 91e07238-9313-b639-00d7-4caf2cd4ff45 [Follower]" leader= testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: added peer, starting replication: peer=server-91e07238-9313-b639-00d7-4caf2cd4ff45 testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.928Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: aborting pipeline replication: peer="{Voter server-1d41a700-ab7c-4948-cf83-b63a833e14fc 1d41a700-ab7c-4948-cf83-b63a833e14fc}" testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: added peer, starting replication: peer=server-91e07238-9313-b639-00d7-4caf2cd4ff45 testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: added peer, starting replication: peer=server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.928Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: added peer, starting replication: peer=server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.928Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: aborting pipeline replication: peer="{Voter server-1d41a700-ab7c-4948-cf83-b63a833e14fc 1d41a700-ab7c-4948-cf83-b63a833e14fc}" testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: pipelining replication: peer="{Voter server-91e07238-9313-b639-00d7-4caf2cd4ff45 91e07238-9313-b639-00d7-4caf2cd4ff45}" testing.go:149: cluster: 2021-08-25T09:28:17.928Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.928Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: aborting pipeline replication: peer="{Voter server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 7b9835ce-b8b3-5c56-a8f5-df64f9056b76}" testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: pipelining replication: peer="{Voter server-91e07238-9313-b639-00d7-4caf2cd4ff45 91e07238-9313-b639-00d7-4caf2cd4ff45}" testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.928Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: aborting pipeline replication: peer="{Voter server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 7b9835ce-b8b3-5c56-a8f5-df64f9056b76}" testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.929Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: pipelining replication: peer="{Voter server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 7b9835ce-b8b3-5c56-a8f5-df64f9056b76}" testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.929Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: pipelining replication: peer="{Voter server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 7b9835ce-b8b3-5c56-a8f5-df64f9056b76}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.034Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.929103372 +0000 UTC m=+17.693319301 (1 nodes), 41.996µs from start of poll, 286.199021ms from cluster start. Timeout at 2021-08-25 09:28:18.034235302 +0000 UTC m=+17.798451270, 105.131969ms after stability testing.go:149: cluster: 2021-08-25T09:28:18.034Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.929103372 +0000 UTC m=+17.693319301 (1 nodes), 41.996µs from start of poll, 286.199021ms from cluster start. Timeout at 2021-08-25 09:28:18.034235302 +0000 UTC m=+17.798451270, 105.131969ms after stability testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:18.034Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: aborting pipeline replication: peer="{Voter server-91e07238-9313-b639-00d7-4caf2cd4ff45 91e07238-9313-b639-00d7-4caf2cd4ff45}" testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:18.034Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: aborting pipeline replication: peer="{Voter server-91e07238-9313-b639-00d7-4caf2cd4ff45 91e07238-9313-b639-00d7-4caf2cd4ff45}" testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:18.034Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: aborting pipeline replication: peer="{Voter server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 7b9835ce-b8b3-5c56-a8f5-df64f9056b76}" testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:18.034Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: aborting pipeline replication: peer="{Voter server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 7b9835ce-b8b3-5c56-a8f5-df64f9056b76}" --- PASS: TestRaft_LeadershipTransfer (0.39s) === RUN TestRaft_LeadershipTransferWithOneNode testLoggerAdapter verbose: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.037Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b78342e0-7634-b610-2d55-bb4ad6b0538b Address:b78342e0-7634-b610-2d55-bb4ad6b0538b}]" testing.go:149: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.037Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b78342e0-7634-b610-2d55-bb4ad6b0538b Address:b78342e0-7634-b610-2d55-bb4ad6b0538b}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.037Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.037Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: entering follower state: follower="Node at b78342e0-7634-b610-2d55-bb4ad6b0538b [Follower]" leader= testing.go:149: cluster: 2021-08-25T09:28:18.037Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.037Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: entering follower state: follower="Node at b78342e0-7634-b610-2d55-bb4ad6b0538b [Follower]" leader= testLoggerAdapter verbose: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.125Z [WARN] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.125Z [WARN] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.126Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: entering candidate state: node="Node at b78342e0-7634-b610-2d55-bb4ad6b0538b [Candidate]" term=2 testing.go:149: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.126Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: entering candidate state: node="Node at b78342e0-7634-b610-2d55-bb4ad6b0538b [Candidate]" term=2 testLoggerAdapter verbose: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.126Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: election won: tally=1 testing.go:149: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.126Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: election won: tally=1 testLoggerAdapter verbose: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.126Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: entering leader state: leader="Node at b78342e0-7634-b610-2d55-bb4ad6b0538b [Leader]" testing.go:149: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.126Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: entering leader state: leader="Node at b78342e0-7634-b610-2d55-bb4ad6b0538b [Leader]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.231Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.126428453 +0000 UTC m=+17.890644342 (1 nodes), 88.966811ms from start of poll, 89.52434ms from cluster start. Timeout at 2021-08-25 09:28:18.231611018 +0000 UTC m=+17.995827007, 105.182665ms after stability testing.go:149: cluster: 2021-08-25T09:28:18.231Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.126428453 +0000 UTC m=+17.890644342 (1 nodes), 88.966811ms from start of poll, 89.52434ms from cluster start. Timeout at 2021-08-25 09:28:18.231611018 +0000 UTC m=+17.995827007, 105.182665ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.20s) === RUN TestRaft_LeadershipTransferWithSevenNodes testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.237Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.237Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" testLoggerAdapter verbose: server-c97ba618-5b89-1423-28c9-da77f6848816: 2021-08-25T09:28:18.237Z [INFO] server-c97ba618-5b89-1423-28c9-da77f6848816: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" testing.go:149: server-c97ba618-5b89-1423-28c9-da77f6848816: 2021-08-25T09:28:18.237Z [INFO] server-c97ba618-5b89-1423-28c9-da77f6848816: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" testLoggerAdapter verbose: server-72bcf709-298a-8071-db2b-49e8c68dc077: 2021-08-25T09:28:18.238Z [INFO] server-72bcf709-298a-8071-db2b-49e8c68dc077: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" testing.go:149: server-72bcf709-298a-8071-db2b-49e8c68dc077: 2021-08-25T09:28:18.238Z [INFO] server-72bcf709-298a-8071-db2b-49e8c68dc077: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" testLoggerAdapter verbose: server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb: 2021-08-25T09:28:18.238Z [INFO] server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" testing.go:149: server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb: 2021-08-25T09:28:18.238Z [INFO] server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" testLoggerAdapter verbose: server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e: 2021-08-25T09:28:18.238Z [INFO] server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" testing.go:149: server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e: 2021-08-25T09:28:18.238Z [INFO] server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.239Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.239Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" testLoggerAdapter verbose: server-353d806d-e21e-17bb-365b-03cccd75b8ef: 2021-08-25T09:28:18.239Z [INFO] server-353d806d-e21e-17bb-365b-03cccd75b8ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" testing.go:149: server-353d806d-e21e-17bb-365b-03cccd75b8ef: 2021-08-25T09:28:18.239Z [INFO] server-353d806d-e21e-17bb-365b-03cccd75b8ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.239Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:18.239Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.239Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: entering follower state: follower="Node at a8082c39-cd11-8752-6816-782956e2cdcf [Follower]" leader= testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.239Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: entering follower state: follower="Node at a8082c39-cd11-8752-6816-782956e2cdcf [Follower]" leader= testLoggerAdapter verbose: server-c97ba618-5b89-1423-28c9-da77f6848816: 2021-08-25T09:28:18.240Z [INFO] server-c97ba618-5b89-1423-28c9-da77f6848816: entering follower state: follower="Node at c97ba618-5b89-1423-28c9-da77f6848816 [Follower]" leader= testing.go:149: server-c97ba618-5b89-1423-28c9-da77f6848816: 2021-08-25T09:28:18.240Z [INFO] server-c97ba618-5b89-1423-28c9-da77f6848816: entering follower state: follower="Node at c97ba618-5b89-1423-28c9-da77f6848816 [Follower]" leader= testLoggerAdapter verbose: server-72bcf709-298a-8071-db2b-49e8c68dc077: 2021-08-25T09:28:18.240Z [INFO] server-72bcf709-298a-8071-db2b-49e8c68dc077: entering follower state: follower="Node at 72bcf709-298a-8071-db2b-49e8c68dc077 [Follower]" leader= testing.go:149: server-72bcf709-298a-8071-db2b-49e8c68dc077: 2021-08-25T09:28:18.240Z [INFO] server-72bcf709-298a-8071-db2b-49e8c68dc077: entering follower state: follower="Node at 72bcf709-298a-8071-db2b-49e8c68dc077 [Follower]" leader= testLoggerAdapter verbose: server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb: 2021-08-25T09:28:18.240Z [INFO] server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb: entering follower state: follower="Node at 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb [Follower]" leader= testing.go:149: server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb: 2021-08-25T09:28:18.240Z [INFO] server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb: entering follower state: follower="Node at 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb [Follower]" leader= testLoggerAdapter verbose: server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e: 2021-08-25T09:28:18.240Z [INFO] server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e: entering follower state: follower="Node at 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e [Follower]" leader= testing.go:149: server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e: 2021-08-25T09:28:18.240Z [INFO] server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e: entering follower state: follower="Node at 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e [Follower]" leader= testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.240Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: entering follower state: follower="Node at 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 [Follower]" leader= testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.240Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: entering follower state: follower="Node at 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 [Follower]" leader= testLoggerAdapter verbose: server-353d806d-e21e-17bb-365b-03cccd75b8ef: 2021-08-25T09:28:18.240Z [INFO] server-353d806d-e21e-17bb-365b-03cccd75b8ef: entering follower state: follower="Node at 353d806d-e21e-17bb-365b-03cccd75b8ef [Follower]" leader= testing.go:149: server-353d806d-e21e-17bb-365b-03cccd75b8ef: 2021-08-25T09:28:18.240Z [INFO] server-353d806d-e21e-17bb-365b-03cccd75b8ef: entering follower state: follower="Node at 353d806d-e21e-17bb-365b-03cccd75b8ef [Follower]" leader= testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.312Z [WARN] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.312Z [WARN] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.312Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: entering candidate state: node="Node at 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 [Candidate]" term=2 testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.312Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: entering candidate state: node="Node at 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 [Candidate]" term=2 testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: election won: tally=4 testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: election won: tally=4 testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: entering leader state: leader="Node at 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 [Leader]" testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: entering leader state: leader="Node at 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 [Leader]" testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-a8082c39-cd11-8752-6816-782956e2cdcf testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-a8082c39-cd11-8752-6816-782956e2cdcf testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-c97ba618-5b89-1423-28c9-da77f6848816 testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-c97ba618-5b89-1423-28c9-da77f6848816 testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-72bcf709-298a-8071-db2b-49e8c68dc077 testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-72bcf709-298a-8071-db2b-49e8c68dc077 testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-a8082c39-cd11-8752-6816-782956e2cdcf a8082c39-cd11-8752-6816-782956e2cdcf}" testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-a8082c39-cd11-8752-6816-782956e2cdcf a8082c39-cd11-8752-6816-782956e2cdcf}" testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-353d806d-e21e-17bb-365b-03cccd75b8ef testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-353d806d-e21e-17bb-365b-03cccd75b8ef testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb}" testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb}" testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.314Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-353d806d-e21e-17bb-365b-03cccd75b8ef 353d806d-e21e-17bb-365b-03cccd75b8ef}" testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.314Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-353d806d-e21e-17bb-365b-03cccd75b8ef 353d806d-e21e-17bb-365b-03cccd75b8ef}" testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-72bcf709-298a-8071-db2b-49e8c68dc077 72bcf709-298a-8071-db2b-49e8c68dc077}" testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-72bcf709-298a-8071-db2b-49e8c68dc077 72bcf709-298a-8071-db2b-49e8c68dc077}" testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.314Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e}" testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.314Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e}" testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-c97ba618-5b89-1423-28c9-da77f6848816 c97ba618-5b89-1423-28c9-da77f6848816}" testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-c97ba618-5b89-1423-28c9-da77f6848816 c97ba618-5b89-1423-28c9-da77f6848816}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.418Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.313232632 +0000 UTC m=+18.077448521 (1 nodes), 73.361862ms from start of poll, 76.259373ms from cluster start. Timeout at 2021-08-25 09:28:18.418416056 +0000 UTC m=+18.182632025, 105.183504ms after stability testing.go:149: cluster: 2021-08-25T09:28:18.418Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.313232632 +0000 UTC m=+18.077448521 (1 nodes), 73.361862ms from start of poll, 76.259373ms from cluster start. Timeout at 2021-08-25 09:28:18.418416056 +0000 UTC m=+18.182632025, 105.183504ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.418Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:18.418Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.523Z [INFO] stable state for Follower reached at 2021-08-25 09:28:18.418678932 +0000 UTC m=+18.182894821 (6 nodes), 3.9µs from start of poll, 181.705673ms from cluster start. Timeout at 2021-08-25 09:28:18.523750287 +0000 UTC m=+18.287966196, 105.071375ms after stability testing.go:149: cluster: 2021-08-25T09:28:18.523Z [INFO] stable state for Follower reached at 2021-08-25 09:28:18.418678932 +0000 UTC m=+18.182894821 (6 nodes), 3.9µs from start of poll, 181.705673ms from cluster start. Timeout at 2021-08-25 09:28:18.523750287 +0000 UTC m=+18.287966196, 105.071375ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.523Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:18.523Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.629Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.52404034 +0000 UTC m=+18.288256229 (1 nodes), 3.48µs from start of poll, 287.067081ms from cluster start. Timeout at 2021-08-25 09:28:18.629165609 +0000 UTC m=+18.393381578, 105.125349ms after stability testing.go:149: cluster: 2021-08-25T09:28:18.629Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.52404034 +0000 UTC m=+18.288256229 (1 nodes), 3.48µs from start of poll, 287.067081ms from cluster start. Timeout at 2021-08-25 09:28:18.629165609 +0000 UTC m=+18.393381578, 105.125349ms after stability testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.629Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: entering candidate state: node="Node at a8082c39-cd11-8752-6816-782956e2cdcf [Candidate]" term=3 testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.629Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: entering candidate state: node="Node at a8082c39-cd11-8752-6816-782956e2cdcf [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.629Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:18.629Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.629Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: entering follower state: follower="Node at 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 [Follower]" leader= testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.629Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: election won: tally=4 testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.629Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: entering follower state: follower="Node at 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 [Follower]" leader= testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.629Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: election won: tally=4 testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.629Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-353d806d-e21e-17bb-365b-03cccd75b8ef 353d806d-e21e-17bb-365b-03cccd75b8ef}" testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.629Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: entering leader state: leader="Node at a8082c39-cd11-8752-6816-782956e2cdcf [Leader]" testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.629Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-353d806d-e21e-17bb-365b-03cccd75b8ef 353d806d-e21e-17bb-365b-03cccd75b8ef}" testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.629Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: entering leader state: leader="Node at a8082c39-cd11-8752-6816-782956e2cdcf [Leader]" testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-c97ba618-5b89-1423-28c9-da77f6848816 testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-c97ba618-5b89-1423-28c9-da77f6848816 testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-72bcf709-298a-8071-db2b-49e8c68dc077 testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-72bcf709-298a-8071-db2b-49e8c68dc077 testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-353d806d-e21e-17bb-365b-03cccd75b8ef testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-353d806d-e21e-17bb-365b-03cccd75b8ef testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-a8082c39-cd11-8752-6816-782956e2cdcf a8082c39-cd11-8752-6816-782956e2cdcf}" testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e}" testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-a8082c39-cd11-8752-6816-782956e2cdcf a8082c39-cd11-8752-6816-782956e2cdcf}" testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb}" testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e}" testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb}" testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6}" testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6}" testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-c97ba618-5b89-1423-28c9-da77f6848816 c97ba618-5b89-1423-28c9-da77f6848816}" testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-c97ba618-5b89-1423-28c9-da77f6848816 c97ba618-5b89-1423-28c9-da77f6848816}" testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-72bcf709-298a-8071-db2b-49e8c68dc077 72bcf709-298a-8071-db2b-49e8c68dc077}" testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-72bcf709-298a-8071-db2b-49e8c68dc077 72bcf709-298a-8071-db2b-49e8c68dc077}" testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-72bcf709-298a-8071-db2b-49e8c68dc077 72bcf709-298a-8071-db2b-49e8c68dc077}" testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-72bcf709-298a-8071-db2b-49e8c68dc077 72bcf709-298a-8071-db2b-49e8c68dc077}" testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-353d806d-e21e-17bb-365b-03cccd75b8ef 353d806d-e21e-17bb-365b-03cccd75b8ef}" testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-353d806d-e21e-17bb-365b-03cccd75b8ef 353d806d-e21e-17bb-365b-03cccd75b8ef}" testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e}" testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e}" testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-c97ba618-5b89-1423-28c9-da77f6848816 c97ba618-5b89-1423-28c9-da77f6848816}" testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-c97ba618-5b89-1423-28c9-da77f6848816 c97ba618-5b89-1423-28c9-da77f6848816}" testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb}" testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.735Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.630038248 +0000 UTC m=+18.394254137 (1 nodes), 159.905µs from start of poll, 393.064989ms from cluster start. Timeout at 2021-08-25 09:28:18.735174797 +0000 UTC m=+18.499390766, 105.136629ms after stability testing.go:149: cluster: 2021-08-25T09:28:18.735Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.630038248 +0000 UTC m=+18.394254137 (1 nodes), 159.905µs from start of poll, 393.064989ms from cluster start. Timeout at 2021-08-25 09:28:18.735174797 +0000 UTC m=+18.499390766, 105.136629ms after stability testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-72bcf709-298a-8071-db2b-49e8c68dc077 72bcf709-298a-8071-db2b-49e8c68dc077}" testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-72bcf709-298a-8071-db2b-49e8c68dc077 72bcf709-298a-8071-db2b-49e8c68dc077}" testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6}" testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6}" testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb}" testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb}" testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e}" testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e}" testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-353d806d-e21e-17bb-365b-03cccd75b8ef 353d806d-e21e-17bb-365b-03cccd75b8ef}" testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-353d806d-e21e-17bb-365b-03cccd75b8ef 353d806d-e21e-17bb-365b-03cccd75b8ef}" testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-c97ba618-5b89-1423-28c9-da77f6848816 c97ba618-5b89-1423-28c9-da77f6848816}" testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-c97ba618-5b89-1423-28c9-da77f6848816 c97ba618-5b89-1423-28c9-da77f6848816}" testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.785Z [ERROR] server-a8082c39-cd11-8752-6816-782956e2cdcf: failed to heartbeat to: peer=29bdfba5-3282-fa54-dbf7-14ad41ce8ebb error="command timed out" testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.785Z [ERROR] server-a8082c39-cd11-8752-6816-782956e2cdcf: failed to heartbeat to: peer=29bdfba5-3282-fa54-dbf7-14ad41ce8ebb error="command timed out" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.56s) === RUN TestRaft_LeadershipTransferToInvalidID testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.795Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c Address:34f0726e-82c8-3ea9-b7cb-ac91ba463d0c} {Suffrage:Voter ID:server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb Address:76da4f5b-7ebc-82a9-60e7-20ec4abda3bb} {Suffrage:Voter ID:server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d Address:78dfc85e-3399-7889-886a-3c2cbd2b8f9d}]" testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.795Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c Address:34f0726e-82c8-3ea9-b7cb-ac91ba463d0c} {Suffrage:Voter ID:server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb Address:76da4f5b-7ebc-82a9-60e7-20ec4abda3bb} {Suffrage:Voter ID:server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d Address:78dfc85e-3399-7889-886a-3c2cbd2b8f9d}]" testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.795Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: entering follower state: follower="Node at 34f0726e-82c8-3ea9-b7cb-ac91ba463d0c [Follower]" leader= testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.795Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: entering follower state: follower="Node at 34f0726e-82c8-3ea9-b7cb-ac91ba463d0c [Follower]" leader= testLoggerAdapter verbose: server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb: 2021-08-25T09:28:18.795Z [INFO] server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c Address:34f0726e-82c8-3ea9-b7cb-ac91ba463d0c} {Suffrage:Voter ID:server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb Address:76da4f5b-7ebc-82a9-60e7-20ec4abda3bb} {Suffrage:Voter ID:server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d Address:78dfc85e-3399-7889-886a-3c2cbd2b8f9d}]" testing.go:149: server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb: 2021-08-25T09:28:18.795Z [INFO] server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c Address:34f0726e-82c8-3ea9-b7cb-ac91ba463d0c} {Suffrage:Voter ID:server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb Address:76da4f5b-7ebc-82a9-60e7-20ec4abda3bb} {Suffrage:Voter ID:server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d Address:78dfc85e-3399-7889-886a-3c2cbd2b8f9d}]" testLoggerAdapter verbose: server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d: 2021-08-25T09:28:18.796Z [INFO] server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c Address:34f0726e-82c8-3ea9-b7cb-ac91ba463d0c} {Suffrage:Voter ID:server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb Address:76da4f5b-7ebc-82a9-60e7-20ec4abda3bb} {Suffrage:Voter ID:server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d Address:78dfc85e-3399-7889-886a-3c2cbd2b8f9d}]" testing.go:149: server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d: 2021-08-25T09:28:18.796Z [INFO] server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c Address:34f0726e-82c8-3ea9-b7cb-ac91ba463d0c} {Suffrage:Voter ID:server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb Address:76da4f5b-7ebc-82a9-60e7-20ec4abda3bb} {Suffrage:Voter ID:server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d Address:78dfc85e-3399-7889-886a-3c2cbd2b8f9d}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.796Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:18.796Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb: 2021-08-25T09:28:18.796Z [INFO] server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb: entering follower state: follower="Node at 76da4f5b-7ebc-82a9-60e7-20ec4abda3bb [Follower]" leader= testing.go:149: server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb: 2021-08-25T09:28:18.796Z [INFO] server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb: entering follower state: follower="Node at 76da4f5b-7ebc-82a9-60e7-20ec4abda3bb [Follower]" leader= testLoggerAdapter verbose: server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d: 2021-08-25T09:28:18.796Z [INFO] server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d: entering follower state: follower="Node at 78dfc85e-3399-7889-886a-3c2cbd2b8f9d [Follower]" leader= testing.go:149: server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d: 2021-08-25T09:28:18.796Z [INFO] server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d: entering follower state: follower="Node at 78dfc85e-3399-7889-886a-3c2cbd2b8f9d [Follower]" leader= testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.860Z [WARN] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.860Z [WARN] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: entering candidate state: node="Node at 34f0726e-82c8-3ea9-b7cb-ac91ba463d0c [Candidate]" term=2 testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: entering candidate state: node="Node at 34f0726e-82c8-3ea9-b7cb-ac91ba463d0c [Candidate]" term=2 testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: election won: tally=2 testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: election won: tally=2 testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: entering leader state: leader="Node at 34f0726e-82c8-3ea9-b7cb-ac91ba463d0c [Leader]" testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: entering leader state: leader="Node at 34f0726e-82c8-3ea9-b7cb-ac91ba463d0c [Leader]" testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: added peer, starting replication: peer=server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: added peer, starting replication: peer=server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: added peer, starting replication: peer=server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: added peer, starting replication: peer=server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: pipelining replication: peer="{Voter server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d 78dfc85e-3399-7889-886a-3c2cbd2b8f9d}" testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: pipelining replication: peer="{Voter server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d 78dfc85e-3399-7889-886a-3c2cbd2b8f9d}" testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.862Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: pipelining replication: peer="{Voter server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb 76da4f5b-7ebc-82a9-60e7-20ec4abda3bb}" testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.862Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: pipelining replication: peer="{Voter server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb 76da4f5b-7ebc-82a9-60e7-20ec4abda3bb}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.967Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.86200483 +0000 UTC m=+18.626220739 (1 nodes), 65.523611ms from start of poll, 66.895943ms from cluster start. Timeout at 2021-08-25 09:28:18.967196413 +0000 UTC m=+18.731412302, 105.191563ms after stability testing.go:149: cluster: 2021-08-25T09:28:18.967Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.86200483 +0000 UTC m=+18.626220739 (1 nodes), 65.523611ms from start of poll, 66.895943ms from cluster start. Timeout at 2021-08-25 09:28:18.967196413 +0000 UTC m=+18.731412302, 105.191563ms after stability testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.967Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: aborting pipeline replication: peer="{Voter server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d 78dfc85e-3399-7889-886a-3c2cbd2b8f9d}" testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.967Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: aborting pipeline replication: peer="{Voter server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d 78dfc85e-3399-7889-886a-3c2cbd2b8f9d}" testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.967Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: aborting pipeline replication: peer="{Voter server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb 76da4f5b-7ebc-82a9-60e7-20ec4abda3bb}" testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.967Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: aborting pipeline replication: peer="{Voter server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb 76da4f5b-7ebc-82a9-60e7-20ec4abda3bb}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.18s) === RUN TestRaft_LeadershipTransferToInvalidAddress testLoggerAdapter verbose: server-f964fe5d-6ec0-adef-29a9-bb2bf093482b: 2021-08-25T09:28:18.971Z [INFO] server-f964fe5d-6ec0-adef-29a9-bb2bf093482b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f964fe5d-6ec0-adef-29a9-bb2bf093482b Address:f964fe5d-6ec0-adef-29a9-bb2bf093482b} {Suffrage:Voter ID:server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb Address:9bb68fd8-217c-79c1-a5ba-b80a222ef6eb} {Suffrage:Voter ID:server-95183a1a-a854-3cf0-1f2a-ec0e624f7351 Address:95183a1a-a854-3cf0-1f2a-ec0e624f7351}]" testing.go:149: server-f964fe5d-6ec0-adef-29a9-bb2bf093482b: 2021-08-25T09:28:18.971Z [INFO] server-f964fe5d-6ec0-adef-29a9-bb2bf093482b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f964fe5d-6ec0-adef-29a9-bb2bf093482b Address:f964fe5d-6ec0-adef-29a9-bb2bf093482b} {Suffrage:Voter ID:server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb Address:9bb68fd8-217c-79c1-a5ba-b80a222ef6eb} {Suffrage:Voter ID:server-95183a1a-a854-3cf0-1f2a-ec0e624f7351 Address:95183a1a-a854-3cf0-1f2a-ec0e624f7351}]" testLoggerAdapter verbose: server-f964fe5d-6ec0-adef-29a9-bb2bf093482b: 2021-08-25T09:28:18.971Z [INFO] server-f964fe5d-6ec0-adef-29a9-bb2bf093482b: entering follower state: follower="Node at f964fe5d-6ec0-adef-29a9-bb2bf093482b [Follower]" leader= testing.go:149: server-f964fe5d-6ec0-adef-29a9-bb2bf093482b: 2021-08-25T09:28:18.971Z [INFO] server-f964fe5d-6ec0-adef-29a9-bb2bf093482b: entering follower state: follower="Node at f964fe5d-6ec0-adef-29a9-bb2bf093482b [Follower]" leader= testLoggerAdapter verbose: server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb: 2021-08-25T09:28:18.971Z [INFO] server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f964fe5d-6ec0-adef-29a9-bb2bf093482b Address:f964fe5d-6ec0-adef-29a9-bb2bf093482b} {Suffrage:Voter ID:server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb Address:9bb68fd8-217c-79c1-a5ba-b80a222ef6eb} {Suffrage:Voter ID:server-95183a1a-a854-3cf0-1f2a-ec0e624f7351 Address:95183a1a-a854-3cf0-1f2a-ec0e624f7351}]" testing.go:149: server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb: 2021-08-25T09:28:18.971Z [INFO] server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f964fe5d-6ec0-adef-29a9-bb2bf093482b Address:f964fe5d-6ec0-adef-29a9-bb2bf093482b} {Suffrage:Voter ID:server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb Address:9bb68fd8-217c-79c1-a5ba-b80a222ef6eb} {Suffrage:Voter ID:server-95183a1a-a854-3cf0-1f2a-ec0e624f7351 Address:95183a1a-a854-3cf0-1f2a-ec0e624f7351}]" testLoggerAdapter verbose: server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb: 2021-08-25T09:28:18.971Z [INFO] server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb: entering follower state: follower="Node at 9bb68fd8-217c-79c1-a5ba-b80a222ef6eb [Follower]" leader= testing.go:149: server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb: 2021-08-25T09:28:18.971Z [INFO] server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb: entering follower state: follower="Node at 9bb68fd8-217c-79c1-a5ba-b80a222ef6eb [Follower]" leader= testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:18.971Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f964fe5d-6ec0-adef-29a9-bb2bf093482b Address:f964fe5d-6ec0-adef-29a9-bb2bf093482b} {Suffrage:Voter ID:server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb Address:9bb68fd8-217c-79c1-a5ba-b80a222ef6eb} {Suffrage:Voter ID:server-95183a1a-a854-3cf0-1f2a-ec0e624f7351 Address:95183a1a-a854-3cf0-1f2a-ec0e624f7351}]" testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:18.971Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f964fe5d-6ec0-adef-29a9-bb2bf093482b Address:f964fe5d-6ec0-adef-29a9-bb2bf093482b} {Suffrage:Voter ID:server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb Address:9bb68fd8-217c-79c1-a5ba-b80a222ef6eb} {Suffrage:Voter ID:server-95183a1a-a854-3cf0-1f2a-ec0e624f7351 Address:95183a1a-a854-3cf0-1f2a-ec0e624f7351}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.971Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:18.971Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:18.971Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: entering follower state: follower="Node at 95183a1a-a854-3cf0-1f2a-ec0e624f7351 [Follower]" leader= testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:18.971Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: entering follower state: follower="Node at 95183a1a-a854-3cf0-1f2a-ec0e624f7351 [Follower]" leader= testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.047Z [WARN] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.047Z [WARN] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.047Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: entering candidate state: node="Node at 95183a1a-a854-3cf0-1f2a-ec0e624f7351 [Candidate]" term=2 testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.047Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: entering candidate state: node="Node at 95183a1a-a854-3cf0-1f2a-ec0e624f7351 [Candidate]" term=2 testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: election won: tally=2 testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: election won: tally=2 testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: entering leader state: leader="Node at 95183a1a-a854-3cf0-1f2a-ec0e624f7351 [Leader]" testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: entering leader state: leader="Node at 95183a1a-a854-3cf0-1f2a-ec0e624f7351 [Leader]" testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: added peer, starting replication: peer=server-f964fe5d-6ec0-adef-29a9-bb2bf093482b testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: added peer, starting replication: peer=server-f964fe5d-6ec0-adef-29a9-bb2bf093482b testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: added peer, starting replication: peer=server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: added peer, starting replication: peer=server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: pipelining replication: peer="{Voter server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb 9bb68fd8-217c-79c1-a5ba-b80a222ef6eb}" testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: pipelining replication: peer="{Voter server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb 9bb68fd8-217c-79c1-a5ba-b80a222ef6eb}" testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: pipelining replication: peer="{Voter server-f964fe5d-6ec0-adef-29a9-bb2bf093482b f964fe5d-6ec0-adef-29a9-bb2bf093482b}" testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: pipelining replication: peer="{Voter server-f964fe5d-6ec0-adef-29a9-bb2bf093482b f964fe5d-6ec0-adef-29a9-bb2bf093482b}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.153Z [INFO] stable state for Follower reached at 2021-08-25 09:28:19.048277438 +0000 UTC m=+18.812493347 (2 nodes), 76.254833ms from start of poll, 77.37053ms from cluster start. Timeout at 2021-08-25 09:28:19.153407987 +0000 UTC m=+18.917623956, 105.130609ms after stability testing.go:149: cluster: 2021-08-25T09:28:19.153Z [INFO] stable state for Follower reached at 2021-08-25 09:28:19.048277438 +0000 UTC m=+18.812493347 (2 nodes), 76.254833ms from start of poll, 77.37053ms from cluster start. Timeout at 2021-08-25 09:28:19.153407987 +0000 UTC m=+18.917623956, 105.130609ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.153Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:19.153Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.258Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.153733857 +0000 UTC m=+18.917949746 (1 nodes), 2.44µs from start of poll, 182.826929ms from cluster start. Timeout at 2021-08-25 09:28:19.258833329 +0000 UTC m=+19.023049278, 105.099532ms after stability testing.go:149: cluster: 2021-08-25T09:28:19.258Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.153733857 +0000 UTC m=+18.917949746 (1 nodes), 2.44µs from start of poll, 182.826929ms from cluster start. Timeout at 2021-08-25 09:28:19.258833329 +0000 UTC m=+19.023049278, 105.099532ms after stability testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.259Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: aborting pipeline replication: peer="{Voter server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb 9bb68fd8-217c-79c1-a5ba-b80a222ef6eb}" testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.259Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: aborting pipeline replication: peer="{Voter server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb 9bb68fd8-217c-79c1-a5ba-b80a222ef6eb}" testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.259Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: aborting pipeline replication: peer="{Voter server-f964fe5d-6ec0-adef-29a9-bb2bf093482b f964fe5d-6ec0-adef-29a9-bb2bf093482b}" testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.259Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: aborting pipeline replication: peer="{Voter server-f964fe5d-6ec0-adef-29a9-bb2bf093482b f964fe5d-6ec0-adef-29a9-bb2bf093482b}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.29s) === RUN TestRaft_LeadershipTransferToBehindServer testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.263Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96cda7d2-fe36-9464-1536-6f6844ad3282 Address:96cda7d2-fe36-9464-1536-6f6844ad3282} {Suffrage:Voter ID:server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf Address:5e246de8-3a20-dc9b-40c0-f4f3fcc616bf} {Suffrage:Voter ID:server-86af87f3-da1d-8428-5c02-ceabc30af9c2 Address:86af87f3-da1d-8428-5c02-ceabc30af9c2}]" testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.263Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96cda7d2-fe36-9464-1536-6f6844ad3282 Address:96cda7d2-fe36-9464-1536-6f6844ad3282} {Suffrage:Voter ID:server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf Address:5e246de8-3a20-dc9b-40c0-f4f3fcc616bf} {Suffrage:Voter ID:server-86af87f3-da1d-8428-5c02-ceabc30af9c2 Address:86af87f3-da1d-8428-5c02-ceabc30af9c2}]" testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.263Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: entering follower state: follower="Node at 96cda7d2-fe36-9464-1536-6f6844ad3282 [Follower]" leader= testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.263Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: entering follower state: follower="Node at 96cda7d2-fe36-9464-1536-6f6844ad3282 [Follower]" leader= testLoggerAdapter verbose: server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf: 2021-08-25T09:28:19.263Z [INFO] server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96cda7d2-fe36-9464-1536-6f6844ad3282 Address:96cda7d2-fe36-9464-1536-6f6844ad3282} {Suffrage:Voter ID:server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf Address:5e246de8-3a20-dc9b-40c0-f4f3fcc616bf} {Suffrage:Voter ID:server-86af87f3-da1d-8428-5c02-ceabc30af9c2 Address:86af87f3-da1d-8428-5c02-ceabc30af9c2}]" testing.go:149: server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf: 2021-08-25T09:28:19.263Z [INFO] server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96cda7d2-fe36-9464-1536-6f6844ad3282 Address:96cda7d2-fe36-9464-1536-6f6844ad3282} {Suffrage:Voter ID:server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf Address:5e246de8-3a20-dc9b-40c0-f4f3fcc616bf} {Suffrage:Voter ID:server-86af87f3-da1d-8428-5c02-ceabc30af9c2 Address:86af87f3-da1d-8428-5c02-ceabc30af9c2}]" testLoggerAdapter verbose: server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf: 2021-08-25T09:28:19.263Z [INFO] server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf: entering follower state: follower="Node at 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf [Follower]" leader= testing.go:149: server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf: 2021-08-25T09:28:19.263Z [INFO] server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf: entering follower state: follower="Node at 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf [Follower]" leader= testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.264Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96cda7d2-fe36-9464-1536-6f6844ad3282 Address:96cda7d2-fe36-9464-1536-6f6844ad3282} {Suffrage:Voter ID:server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf Address:5e246de8-3a20-dc9b-40c0-f4f3fcc616bf} {Suffrage:Voter ID:server-86af87f3-da1d-8428-5c02-ceabc30af9c2 Address:86af87f3-da1d-8428-5c02-ceabc30af9c2}]" testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.264Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96cda7d2-fe36-9464-1536-6f6844ad3282 Address:96cda7d2-fe36-9464-1536-6f6844ad3282} {Suffrage:Voter ID:server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf Address:5e246de8-3a20-dc9b-40c0-f4f3fcc616bf} {Suffrage:Voter ID:server-86af87f3-da1d-8428-5c02-ceabc30af9c2 Address:86af87f3-da1d-8428-5c02-ceabc30af9c2}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.264Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.264Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: entering follower state: follower="Node at 86af87f3-da1d-8428-5c02-ceabc30af9c2 [Follower]" leader= testing.go:149: cluster: 2021-08-25T09:28:19.264Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.264Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: entering follower state: follower="Node at 86af87f3-da1d-8428-5c02-ceabc30af9c2 [Follower]" leader= testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.328Z [WARN] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.328Z [WARN] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.328Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: entering candidate state: node="Node at 86af87f3-da1d-8428-5c02-ceabc30af9c2 [Candidate]" term=2 testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.328Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: entering candidate state: node="Node at 86af87f3-da1d-8428-5c02-ceabc30af9c2 [Candidate]" term=2 testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.328Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: election won: tally=2 testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.328Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: election won: tally=2 testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.328Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: entering leader state: leader="Node at 86af87f3-da1d-8428-5c02-ceabc30af9c2 [Leader]" testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.328Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: entering leader state: leader="Node at 86af87f3-da1d-8428-5c02-ceabc30af9c2 [Leader]" testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.329Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: added peer, starting replication: peer=server-96cda7d2-fe36-9464-1536-6f6844ad3282 testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.329Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: added peer, starting replication: peer=server-96cda7d2-fe36-9464-1536-6f6844ad3282 testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.329Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: added peer, starting replication: peer=server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.329Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: added peer, starting replication: peer=server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.329Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: pipelining replication: peer="{Voter server-96cda7d2-fe36-9464-1536-6f6844ad3282 96cda7d2-fe36-9464-1536-6f6844ad3282}" testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.329Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: pipelining replication: peer="{Voter server-96cda7d2-fe36-9464-1536-6f6844ad3282 96cda7d2-fe36-9464-1536-6f6844ad3282}" testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.329Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: pipelining replication: peer="{Voter server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf}" testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.329Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: pipelining replication: peer="{Voter server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.434Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.329094759 +0000 UTC m=+19.093310648 (1 nodes), 64.655971ms from start of poll, 65.970649ms from cluster start. Timeout at 2021-08-25 09:28:19.434246306 +0000 UTC m=+19.198462215, 105.151567ms after stability testing.go:149: cluster: 2021-08-25T09:28:19.434Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.329094759 +0000 UTC m=+19.093310648 (1 nodes), 64.655971ms from start of poll, 65.970649ms from cluster start. Timeout at 2021-08-25 09:28:19.434246306 +0000 UTC m=+19.198462215, 105.151567ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.434Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:19.434Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.539Z [INFO] stable state for Follower reached at 2021-08-25 09:28:19.43452714 +0000 UTC m=+19.198743049 (2 nodes), 2.78µs from start of poll, 171.40305ms from cluster start. Timeout at 2021-08-25 09:28:19.539688307 +0000 UTC m=+19.303904276, 105.161227ms after stability testing.go:149: cluster: 2021-08-25T09:28:19.539Z [INFO] stable state for Follower reached at 2021-08-25 09:28:19.43452714 +0000 UTC m=+19.198743049 (2 nodes), 2.78µs from start of poll, 171.40305ms from cluster start. Timeout at 2021-08-25 09:28:19.539688307 +0000 UTC m=+19.303904276, 105.161227ms after stability testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.578Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: entering candidate state: node="Node at 96cda7d2-fe36-9464-1536-6f6844ad3282 [Candidate]" term=3 testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.578Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: entering candidate state: node="Node at 96cda7d2-fe36-9464-1536-6f6844ad3282 [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.578Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.578Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: entering follower state: follower="Node at 86af87f3-da1d-8428-5c02-ceabc30af9c2 [Follower]" leader= testing.go:149: cluster: 2021-08-25T09:28:19.578Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.578Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: election won: tally=2 testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.578Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: entering follower state: follower="Node at 86af87f3-da1d-8428-5c02-ceabc30af9c2 [Follower]" leader= testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.578Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: aborting pipeline replication: peer="{Voter server-96cda7d2-fe36-9464-1536-6f6844ad3282 96cda7d2-fe36-9464-1536-6f6844ad3282}" testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.578Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: election won: tally=2 testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.578Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: entering leader state: leader="Node at 96cda7d2-fe36-9464-1536-6f6844ad3282 [Leader]" testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.578Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: aborting pipeline replication: peer="{Voter server-96cda7d2-fe36-9464-1536-6f6844ad3282 96cda7d2-fe36-9464-1536-6f6844ad3282}" testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.578Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: entering leader state: leader="Node at 96cda7d2-fe36-9464-1536-6f6844ad3282 [Leader]" testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.578Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: aborting pipeline replication: peer="{Voter server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf}" testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.578Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: added peer, starting replication: peer=server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.578Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: aborting pipeline replication: peer="{Voter server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf}" testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.578Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: added peer, starting replication: peer=server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.579Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: added peer, starting replication: peer=server-86af87f3-da1d-8428-5c02-ceabc30af9c2 testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.579Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: added peer, starting replication: peer=server-86af87f3-da1d-8428-5c02-ceabc30af9c2 testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.579Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: pipelining replication: peer="{Voter server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf}" testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.579Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: pipelining replication: peer="{Voter server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf}" testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.579Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: pipelining replication: peer="{Voter server-86af87f3-da1d-8428-5c02-ceabc30af9c2 86af87f3-da1d-8428-5c02-ceabc30af9c2}" testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.579Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: pipelining replication: peer="{Voter server-86af87f3-da1d-8428-5c02-ceabc30af9c2 86af87f3-da1d-8428-5c02-ceabc30af9c2}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.684Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.57881843 +0000 UTC m=+19.343034319 (1 nodes), 291.713µs from start of poll, 315.69432ms from cluster start. Timeout at 2021-08-25 09:28:19.684143501 +0000 UTC m=+19.448359450, 105.325131ms after stability testing.go:149: cluster: 2021-08-25T09:28:19.684Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.57881843 +0000 UTC m=+19.343034319 (1 nodes), 291.713µs from start of poll, 315.69432ms from cluster start. Timeout at 2021-08-25 09:28:19.684143501 +0000 UTC m=+19.448359450, 105.325131ms after stability testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.684Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: aborting pipeline replication: peer="{Voter server-86af87f3-da1d-8428-5c02-ceabc30af9c2 86af87f3-da1d-8428-5c02-ceabc30af9c2}" testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.684Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: aborting pipeline replication: peer="{Voter server-86af87f3-da1d-8428-5c02-ceabc30af9c2 86af87f3-da1d-8428-5c02-ceabc30af9c2}" testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.684Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: aborting pipeline replication: peer="{Voter server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf}" testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.684Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: aborting pipeline replication: peer="{Voter server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.43s) === RUN TestRaft_LeadershipTransferToItself testLoggerAdapter verbose: server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab: 2021-08-25T09:28:19.688Z [INFO] server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab Address:972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab} {Suffrage:Voter ID:server-532e826a-77e4-388a-9282-45151c6bf272 Address:532e826a-77e4-388a-9282-45151c6bf272} {Suffrage:Voter ID:server-922da962-4151-f9f1-4842-3cc8cada4fa5 Address:922da962-4151-f9f1-4842-3cc8cada4fa5}]" testing.go:149: server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab: 2021-08-25T09:28:19.688Z [INFO] server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab Address:972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab} {Suffrage:Voter ID:server-532e826a-77e4-388a-9282-45151c6bf272 Address:532e826a-77e4-388a-9282-45151c6bf272} {Suffrage:Voter ID:server-922da962-4151-f9f1-4842-3cc8cada4fa5 Address:922da962-4151-f9f1-4842-3cc8cada4fa5}]" testLoggerAdapter verbose: server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab: 2021-08-25T09:28:19.689Z [INFO] server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab: entering follower state: follower="Node at 972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab [Follower]" leader= testing.go:149: server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab: 2021-08-25T09:28:19.689Z [INFO] server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab: entering follower state: follower="Node at 972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab [Follower]" leader= testLoggerAdapter verbose: server-532e826a-77e4-388a-9282-45151c6bf272: 2021-08-25T09:28:19.689Z [INFO] server-532e826a-77e4-388a-9282-45151c6bf272: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab Address:972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab} {Suffrage:Voter ID:server-532e826a-77e4-388a-9282-45151c6bf272 Address:532e826a-77e4-388a-9282-45151c6bf272} {Suffrage:Voter ID:server-922da962-4151-f9f1-4842-3cc8cada4fa5 Address:922da962-4151-f9f1-4842-3cc8cada4fa5}]" testing.go:149: server-532e826a-77e4-388a-9282-45151c6bf272: 2021-08-25T09:28:19.689Z [INFO] server-532e826a-77e4-388a-9282-45151c6bf272: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab Address:972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab} {Suffrage:Voter ID:server-532e826a-77e4-388a-9282-45151c6bf272 Address:532e826a-77e4-388a-9282-45151c6bf272} {Suffrage:Voter ID:server-922da962-4151-f9f1-4842-3cc8cada4fa5 Address:922da962-4151-f9f1-4842-3cc8cada4fa5}]" testLoggerAdapter verbose: server-532e826a-77e4-388a-9282-45151c6bf272: 2021-08-25T09:28:19.689Z [INFO] server-532e826a-77e4-388a-9282-45151c6bf272: entering follower state: follower="Node at 532e826a-77e4-388a-9282-45151c6bf272 [Follower]" leader= testing.go:149: server-532e826a-77e4-388a-9282-45151c6bf272: 2021-08-25T09:28:19.689Z [INFO] server-532e826a-77e4-388a-9282-45151c6bf272: entering follower state: follower="Node at 532e826a-77e4-388a-9282-45151c6bf272 [Follower]" leader= testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.689Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab Address:972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab} {Suffrage:Voter ID:server-532e826a-77e4-388a-9282-45151c6bf272 Address:532e826a-77e4-388a-9282-45151c6bf272} {Suffrage:Voter ID:server-922da962-4151-f9f1-4842-3cc8cada4fa5 Address:922da962-4151-f9f1-4842-3cc8cada4fa5}]" testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.689Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab Address:972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab} {Suffrage:Voter ID:server-532e826a-77e4-388a-9282-45151c6bf272 Address:532e826a-77e4-388a-9282-45151c6bf272} {Suffrage:Voter ID:server-922da962-4151-f9f1-4842-3cc8cada4fa5 Address:922da962-4151-f9f1-4842-3cc8cada4fa5}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.689Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:19.689Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.689Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: entering follower state: follower="Node at 922da962-4151-f9f1-4842-3cc8cada4fa5 [Follower]" leader= testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.689Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: entering follower state: follower="Node at 922da962-4151-f9f1-4842-3cc8cada4fa5 [Follower]" leader= testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [WARN] server-922da962-4151-f9f1-4842-3cc8cada4fa5: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [WARN] server-922da962-4151-f9f1-4842-3cc8cada4fa5: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: entering candidate state: node="Node at 922da962-4151-f9f1-4842-3cc8cada4fa5 [Candidate]" term=2 testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: entering candidate state: node="Node at 922da962-4151-f9f1-4842-3cc8cada4fa5 [Candidate]" term=2 testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: election won: tally=2 testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: election won: tally=2 testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: entering leader state: leader="Node at 922da962-4151-f9f1-4842-3cc8cada4fa5 [Leader]" testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: entering leader state: leader="Node at 922da962-4151-f9f1-4842-3cc8cada4fa5 [Leader]" testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: added peer, starting replication: peer=server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: added peer, starting replication: peer=server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: added peer, starting replication: peer=server-532e826a-77e4-388a-9282-45151c6bf272 testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: added peer, starting replication: peer=server-532e826a-77e4-388a-9282-45151c6bf272 testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: pipelining replication: peer="{Voter server-532e826a-77e4-388a-9282-45151c6bf272 532e826a-77e4-388a-9282-45151c6bf272}" testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: pipelining replication: peer="{Voter server-532e826a-77e4-388a-9282-45151c6bf272 532e826a-77e4-388a-9282-45151c6bf272}" testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: pipelining replication: peer="{Voter server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab 972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab}" testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: pipelining replication: peer="{Voter server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab 972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.846Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.740711584 +0000 UTC m=+19.504927473 (1 nodes), 50.714907ms from start of poll, 52.172471ms from cluster start. Timeout at 2021-08-25 09:28:19.84587445 +0000 UTC m=+19.610090419, 105.162946ms after stability testing.go:149: cluster: 2021-08-25T09:28:19.846Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.740711584 +0000 UTC m=+19.504927473 (1 nodes), 50.714907ms from start of poll, 52.172471ms from cluster start. Timeout at 2021-08-25 09:28:19.84587445 +0000 UTC m=+19.610090419, 105.162946ms after stability testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.846Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: cannot transfer leadership to itself testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.846Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: cannot transfer leadership to itself testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.846Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: aborting pipeline replication: peer="{Voter server-532e826a-77e4-388a-9282-45151c6bf272 532e826a-77e4-388a-9282-45151c6bf272}" testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.846Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: aborting pipeline replication: peer="{Voter server-532e826a-77e4-388a-9282-45151c6bf272 532e826a-77e4-388a-9282-45151c6bf272}" testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.846Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: aborting pipeline replication: peer="{Voter server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab 972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab}" testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.846Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: aborting pipeline replication: peer="{Voter server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab 972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab}" --- PASS: TestRaft_LeadershipTransferToItself (0.16s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests raft_test.go:2064: Disabled unreliable test --- SKIP: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.00s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout testLoggerAdapter verbose: server-c2171324-e927-bb2b-3893-ea136cd96306: 2021-08-25T09:28:19.851Z [INFO] server-c2171324-e927-bb2b-3893-ea136cd96306: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2171324-e927-bb2b-3893-ea136cd96306 Address:c2171324-e927-bb2b-3893-ea136cd96306} {Suffrage:Voter ID:server-17720e6c-540b-3eb4-44e0-d74473b8ec95 Address:17720e6c-540b-3eb4-44e0-d74473b8ec95} {Suffrage:Voter ID:server-0b037b32-f225-08b7-e7e3-10aa57b38c0d Address:0b037b32-f225-08b7-e7e3-10aa57b38c0d}]" testing.go:149: server-c2171324-e927-bb2b-3893-ea136cd96306: 2021-08-25T09:28:19.851Z [INFO] server-c2171324-e927-bb2b-3893-ea136cd96306: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2171324-e927-bb2b-3893-ea136cd96306 Address:c2171324-e927-bb2b-3893-ea136cd96306} {Suffrage:Voter ID:server-17720e6c-540b-3eb4-44e0-d74473b8ec95 Address:17720e6c-540b-3eb4-44e0-d74473b8ec95} {Suffrage:Voter ID:server-0b037b32-f225-08b7-e7e3-10aa57b38c0d Address:0b037b32-f225-08b7-e7e3-10aa57b38c0d}]" testLoggerAdapter verbose: server-c2171324-e927-bb2b-3893-ea136cd96306: 2021-08-25T09:28:19.851Z [INFO] server-c2171324-e927-bb2b-3893-ea136cd96306: entering follower state: follower="Node at c2171324-e927-bb2b-3893-ea136cd96306 [Follower]" leader= testing.go:149: server-c2171324-e927-bb2b-3893-ea136cd96306: 2021-08-25T09:28:19.851Z [INFO] server-c2171324-e927-bb2b-3893-ea136cd96306: entering follower state: follower="Node at c2171324-e927-bb2b-3893-ea136cd96306 [Follower]" leader= testLoggerAdapter verbose: server-17720e6c-540b-3eb4-44e0-d74473b8ec95: 2021-08-25T09:28:19.851Z [INFO] server-17720e6c-540b-3eb4-44e0-d74473b8ec95: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2171324-e927-bb2b-3893-ea136cd96306 Address:c2171324-e927-bb2b-3893-ea136cd96306} {Suffrage:Voter ID:server-17720e6c-540b-3eb4-44e0-d74473b8ec95 Address:17720e6c-540b-3eb4-44e0-d74473b8ec95} {Suffrage:Voter ID:server-0b037b32-f225-08b7-e7e3-10aa57b38c0d Address:0b037b32-f225-08b7-e7e3-10aa57b38c0d}]" testing.go:149: server-17720e6c-540b-3eb4-44e0-d74473b8ec95: 2021-08-25T09:28:19.851Z [INFO] server-17720e6c-540b-3eb4-44e0-d74473b8ec95: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2171324-e927-bb2b-3893-ea136cd96306 Address:c2171324-e927-bb2b-3893-ea136cd96306} {Suffrage:Voter ID:server-17720e6c-540b-3eb4-44e0-d74473b8ec95 Address:17720e6c-540b-3eb4-44e0-d74473b8ec95} {Suffrage:Voter ID:server-0b037b32-f225-08b7-e7e3-10aa57b38c0d Address:0b037b32-f225-08b7-e7e3-10aa57b38c0d}]" testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.851Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2171324-e927-bb2b-3893-ea136cd96306 Address:c2171324-e927-bb2b-3893-ea136cd96306} {Suffrage:Voter ID:server-17720e6c-540b-3eb4-44e0-d74473b8ec95 Address:17720e6c-540b-3eb4-44e0-d74473b8ec95} {Suffrage:Voter ID:server-0b037b32-f225-08b7-e7e3-10aa57b38c0d Address:0b037b32-f225-08b7-e7e3-10aa57b38c0d}]" testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.851Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2171324-e927-bb2b-3893-ea136cd96306 Address:c2171324-e927-bb2b-3893-ea136cd96306} {Suffrage:Voter ID:server-17720e6c-540b-3eb4-44e0-d74473b8ec95 Address:17720e6c-540b-3eb4-44e0-d74473b8ec95} {Suffrage:Voter ID:server-0b037b32-f225-08b7-e7e3-10aa57b38c0d Address:0b037b32-f225-08b7-e7e3-10aa57b38c0d}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.851Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:19.851Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-17720e6c-540b-3eb4-44e0-d74473b8ec95: 2021-08-25T09:28:19.852Z [INFO] server-17720e6c-540b-3eb4-44e0-d74473b8ec95: entering follower state: follower="Node at 17720e6c-540b-3eb4-44e0-d74473b8ec95 [Follower]" leader= testing.go:149: server-17720e6c-540b-3eb4-44e0-d74473b8ec95: 2021-08-25T09:28:19.852Z [INFO] server-17720e6c-540b-3eb4-44e0-d74473b8ec95: entering follower state: follower="Node at 17720e6c-540b-3eb4-44e0-d74473b8ec95 [Follower]" leader= testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.852Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: entering follower state: follower="Node at 0b037b32-f225-08b7-e7e3-10aa57b38c0d [Follower]" leader= testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.852Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: entering follower state: follower="Node at 0b037b32-f225-08b7-e7e3-10aa57b38c0d [Follower]" leader= testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.921Z [WARN] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.921Z [WARN] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.921Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: entering candidate state: node="Node at 0b037b32-f225-08b7-e7e3-10aa57b38c0d [Candidate]" term=2 testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.921Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: entering candidate state: node="Node at 0b037b32-f225-08b7-e7e3-10aa57b38c0d [Candidate]" term=2 testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.921Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: election won: tally=2 testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.921Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: election won: tally=2 testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.921Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: entering leader state: leader="Node at 0b037b32-f225-08b7-e7e3-10aa57b38c0d [Leader]" testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.921Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: entering leader state: leader="Node at 0b037b32-f225-08b7-e7e3-10aa57b38c0d [Leader]" testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.922Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: added peer, starting replication: peer=server-c2171324-e927-bb2b-3893-ea136cd96306 testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.922Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: added peer, starting replication: peer=server-c2171324-e927-bb2b-3893-ea136cd96306 testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.922Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: added peer, starting replication: peer=server-17720e6c-540b-3eb4-44e0-d74473b8ec95 testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.922Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: added peer, starting replication: peer=server-17720e6c-540b-3eb4-44e0-d74473b8ec95 testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.922Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: pipelining replication: peer="{Voter server-17720e6c-540b-3eb4-44e0-d74473b8ec95 17720e6c-540b-3eb4-44e0-d74473b8ec95}" testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.922Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: pipelining replication: peer="{Voter server-17720e6c-540b-3eb4-44e0-d74473b8ec95 17720e6c-540b-3eb4-44e0-d74473b8ec95}" testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.922Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: pipelining replication: peer="{Voter server-c2171324-e927-bb2b-3893-ea136cd96306 c2171324-e927-bb2b-3893-ea136cd96306}" testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.922Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: pipelining replication: peer="{Voter server-c2171324-e927-bb2b-3893-ea136cd96306 c2171324-e927-bb2b-3893-ea136cd96306}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:20.027Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.922468052 +0000 UTC m=+19.686683941 (1 nodes), 70.37324ms from start of poll, 71.736453ms from cluster start. Timeout at 2021-08-25 09:28:20.027613719 +0000 UTC m=+19.791829688, 105.145747ms after stability testing.go:149: cluster: 2021-08-25T09:28:20.027Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.922468052 +0000 UTC m=+19.686683941 (1 nodes), 70.37324ms from start of poll, 71.736453ms from cluster start. Timeout at 2021-08-25 09:28:20.027613719 +0000 UTC m=+19.791829688, 105.145747ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:20.027Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:20.027Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-08-25T09:28:20.133Z [INFO] stable state for Follower reached at 2021-08-25 09:28:20.027880535 +0000 UTC m=+19.792096424 (2 nodes), 2.54µs from start of poll, 177.148936ms from cluster start. Timeout at 2021-08-25 09:28:20.132998845 +0000 UTC m=+19.897214834, 105.11841ms after stability testing.go:149: cluster: 2021-08-25T09:28:20.133Z [INFO] stable state for Follower reached at 2021-08-25 09:28:20.027880535 +0000 UTC m=+19.792096424 (2 nodes), 2.54µs from start of poll, 177.148936ms from cluster start. Timeout at 2021-08-25 09:28:20.132998845 +0000 UTC m=+19.897214834, 105.11841ms after stability testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:20.566Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: aborting pipeline replication: peer="{Voter server-c2171324-e927-bb2b-3893-ea136cd96306 c2171324-e927-bb2b-3893-ea136cd96306}" testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:20.566Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: aborting pipeline replication: peer="{Voter server-c2171324-e927-bb2b-3893-ea136cd96306 c2171324-e927-bb2b-3893-ea136cd96306}" testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:20.566Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: aborting pipeline replication: peer="{Voter server-17720e6c-540b-3eb4-44e0-d74473b8ec95 17720e6c-540b-3eb4-44e0-d74473b8ec95}" testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:20.566Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: aborting pipeline replication: peer="{Voter server-17720e6c-540b-3eb4-44e0-d74473b8ec95 17720e6c-540b-3eb4-44e0-d74473b8ec95}" testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:20.616Z [ERROR] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: failed to heartbeat to: peer=17720e6c-540b-3eb4-44e0-d74473b8ec95 error="command timed out" testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:20.616Z [ERROR] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: failed to heartbeat to: peer=17720e6c-540b-3eb4-44e0-d74473b8ec95 error="command timed out" --- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.77s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.620Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 Address:7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2} {Suffrage:Voter ID:server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 Address:27b8f23c-1f52-9be4-0647-d3a0e8dec358}]" testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.620Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 Address:7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2} {Suffrage:Voter ID:server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 Address:27b8f23c-1f52-9be4-0647-d3a0e8dec358}]" testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.621Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: entering follower state: follower="Node at 7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 [Follower]" leader= testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.621Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: entering follower state: follower="Node at 7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 [Follower]" leader= testLoggerAdapter verbose: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.621Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 Address:7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2} {Suffrage:Voter ID:server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 Address:27b8f23c-1f52-9be4-0647-d3a0e8dec358}]" testing.go:149: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.621Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 Address:7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2} {Suffrage:Voter ID:server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 Address:27b8f23c-1f52-9be4-0647-d3a0e8dec358}]" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:20.621Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-08-25T09:28:20.621Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.621Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: entering follower state: follower="Node at 27b8f23c-1f52-9be4-0647-d3a0e8dec358 [Follower]" leader= testing.go:149: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.621Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: entering follower state: follower="Node at 27b8f23c-1f52-9be4-0647-d3a0e8dec358 [Follower]" leader= testLoggerAdapter verbose: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.698Z [WARN] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.698Z [WARN] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.698Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: entering candidate state: node="Node at 27b8f23c-1f52-9be4-0647-d3a0e8dec358 [Candidate]" term=2 testing.go:149: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.698Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: entering candidate state: node="Node at 27b8f23c-1f52-9be4-0647-d3a0e8dec358 [Candidate]" term=2 testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.698Z [WARN] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.698Z [WARN] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.699Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: entering candidate state: node="Node at 7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 [Candidate]" term=2 testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.699Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: entering candidate state: node="Node at 7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 [Candidate]" term=2 testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.699Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: duplicate requestVote for same term: term=2 testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.699Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.699Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: duplicate requestVote for same term: term=2 testing.go:149: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.699Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [WARN] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: Election timeout reached, restarting election testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [WARN] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: Election timeout reached, restarting election testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: entering candidate state: node="Node at 7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 [Candidate]" term=3 testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: entering candidate state: node="Node at 7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 [Candidate]" term=3 testLoggerAdapter verbose: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.755Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: entering follower state: follower="Node at 27b8f23c-1f52-9be4-0647-d3a0e8dec358 [Follower]" leader= testing.go:149: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.755Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: entering follower state: follower="Node at 27b8f23c-1f52-9be4-0647-d3a0e8dec358 [Follower]" leader= testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: election won: tally=2 testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: election won: tally=2 testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: entering leader state: leader="Node at 7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 [Leader]" testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: entering leader state: leader="Node at 7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 [Leader]" testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: added peer, starting replication: peer=server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: added peer, starting replication: peer=server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.756Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: pipelining replication: peer="{Voter server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 27b8f23c-1f52-9be4-0647-d3a0e8dec358}" testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.756Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: pipelining replication: peer="{Voter server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 27b8f23c-1f52-9be4-0647-d3a0e8dec358}" testLoggerAdapter verbose: cluster: 2021-08-25T09:28:20.861Z [INFO] stable state for Follower reached at 2021-08-25 09:28:20.755932431 +0000 UTC m=+20.520148320 (1 nodes), 134.026444ms from start of poll, 135.295946ms from cluster start. Timeout at 2021-08-25 09:28:20.861053341 +0000 UTC m=+20.625269310, 105.12099ms after stability testing.go:149: cluster: 2021-08-25T09:28:20.861Z [INFO] stable state for Follower reached at 2021-08-25 09:28:20.755932431 +0000 UTC m=+20.520148320 (1 nodes), 134.026444ms from start of poll, 135.295946ms from cluster start. Timeout at 2021-08-25 09:28:20.861053341 +0000 UTC m=+20.625269310, 105.12099ms after stability testLoggerAdapter verbose: cluster: 2021-08-25T09:28:20.861Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-08-25T09:28:20.861Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-08-25T09:28:20.966Z [INFO] stable state for Leader reached at 2021-08-25 09:28:20.86148872 +0000 UTC m=+20.625704609 (1 nodes), 2.239µs from start of poll, 240.852235ms from cluster start. Timeout at 2021-08-25 09:28:20.966610131 +0000 UTC m=+20.730826060, 105.121451ms after stability testing.go:149: cluster: 2021-08-25T09:28:20.966Z [INFO] stable state for Leader reached at 2021-08-25 09:28:20.86148872 +0000 UTC m=+20.625704609 (1 nodes), 2.239µs from start of poll, 240.852235ms from cluster start. Timeout at 2021-08-25 09:28:20.966610131 +0000 UTC m=+20.730826060, 105.121451ms after stability testLoggerAdapter verbose: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.977Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 Address:7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2} {Suffrage:Voter ID:server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 Address:27b8f23c-1f52-9be4-0647-d3a0e8dec358}]" testing.go:149: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.977Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 Address:7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2} {Suffrage:Voter ID:server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 Address:27b8f23c-1f52-9be4-0647-d3a0e8dec358}]" testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.977Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: aborting pipeline replication: peer="{Voter server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 27b8f23c-1f52-9be4-0647-d3a0e8dec358}" testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.977Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: aborting pipeline replication: peer="{Voter server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 27b8f23c-1f52-9be4-0647-d3a0e8dec358}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.36s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_WithAdvertise --- PASS: TestTCPTransport_WithAdvertise (0.00s) === RUN TestTransport_StartStop --- PASS: TestTransport_StartStop (0.00s) === RUN TestTransport_AppendEntries --- PASS: TestTransport_AppendEntries (0.00s) === RUN TestTransport_AppendEntriesPipeline --- PASS: TestTransport_AppendEntriesPipeline (0.00s) === RUN TestTransport_RequestVote --- PASS: TestTransport_RequestVote (0.00s) === RUN TestTransport_InstallSnapshot --- PASS: TestTransport_InstallSnapshot (0.00s) === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout --- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed --- PASS: TestNewSeed (0.00s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin --- PASS: TestMin (0.00s) === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID --- PASS: TestGenerateUUID (0.00s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) PASS ok github.com/hashicorp/raft 20.764s ? github.com/hashicorp/raft/bench [no test files] create-stamp debian/debhelper-build-stamp fakeroot debian/rules binary dh binary --buildsystem=golang --with=golang dh_testroot -O--buildsystem=golang dh_prep -O--buildsystem=golang dh_auto_install -O--buildsystem=golang dh_installdocs -O--buildsystem=golang dh_installchangelogs -O--buildsystem=golang dh_installinit -O--buildsystem=golang dh_installsystemduser -O--buildsystem=golang dh_perl -O--buildsystem=golang dh_link -O--buildsystem=golang dh_strip_nondeterminism -O--buildsystem=golang dh_compress -O--buildsystem=golang dh_fixperms -O--buildsystem=golang dh_missing -O--buildsystem=golang dh_installdeb -O--buildsystem=golang dh_golang -O--buildsystem=golang dh_gencontrol -O--buildsystem=golang dpkg-gencontrol: warning: Depends field of package golang-github-hashicorp-raft-dev: substitution variable ${shlibs:Depends} used, but is not defined dh_md5sums -O--buildsystem=golang dh_builddeb -O--buildsystem=golang dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.1.2-1_all.deb'. dpkg-genbuildinfo --build=binary dpkg-genchanges --build=binary >../golang-github-hashicorp-raft_1.1.2-1_armhf.changes dpkg-genchanges: info: binary-only upload (no source code included) dpkg-source --after-build . dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env I: removing directory /srv/workspace/pbuilder/23540 and its subdirectories I: Current time: Tue Aug 24 21:28:32 -12 2021 I: pbuilder-time-stamp: 1629883712 Wed Aug 25 09:28:48 UTC 2021 I: 1st build successful. Starting 2nd build on remote node virt32b-armhf-rb.debian.net. Wed Aug 25 09:28:48 UTC 2021 I: Preparing to do remote build '2' on virt32b-armhf-rb.debian.net. Wed Aug 25 09:31:36 UTC 2021 I: Deleting $TMPDIR on virt32b-armhf-rb.debian.net. Wed Aug 25 09:31:38 UTC 2021 I: golang-github-hashicorp-raft_1.1.2-1_armhf.changes: Format: 1.8 Date: Sun, 19 Jan 2020 18:15:42 +1100 Source: golang-github-hashicorp-raft Binary: golang-github-hashicorp-raft-dev Architecture: all Version: 1.1.2-1 Distribution: unstable Urgency: medium Maintainer: Debian Go Packaging Team Changed-By: Dmitry Smirnov Description: golang-github-hashicorp-raft-dev - implementation of the Raft consensus protocol Changes: golang-github-hashicorp-raft (1.1.2-1) unstable; urgency=medium . * New upstream release. Checksums-Sha1: fd2f48e7cbf9067cdcc7171212e616e619053c70 90896 golang-github-hashicorp-raft-dev_1.1.2-1_all.deb 7c9b77720d9b8755edb07b7dce65713a3b45b842 8572 golang-github-hashicorp-raft_1.1.2-1_armhf.buildinfo Checksums-Sha256: f34877bf2e6d361137752e477ed94bb92556fd43a6ed29fa6135cc010f7d1954 90896 golang-github-hashicorp-raft-dev_1.1.2-1_all.deb 04e600859479f576e5ce223ac7bbbc526af582af7e5453369fb3eae57ef197f2 8572 golang-github-hashicorp-raft_1.1.2-1_armhf.buildinfo Files: 9217d71a882e9745b40d8acfe32cc432 90896 devel optional golang-github-hashicorp-raft-dev_1.1.2-1_all.deb 1d14060f7082f766299c1a121ea9e3e1 8572 devel optional golang-github-hashicorp-raft_1.1.2-1_armhf.buildinfo Wed Aug 25 09:31:40 UTC 2021 I: diffoscope 177 will be used to compare the two builds: # Profiling output for: /usr/bin/diffoscope --html /srv/reproducible-results/rbuild-debian/tmp.wjmIrRuTSj/golang-github-hashicorp-raft_1.1.2-1.diffoscope.html --text /srv/reproducible-results/rbuild-debian/tmp.wjmIrRuTSj/golang-github-hashicorp-raft_1.1.2-1.diffoscope.txt --json /srv/reproducible-results/rbuild-debian/tmp.wjmIrRuTSj/golang-github-hashicorp-raft_1.1.2-1.diffoscope.json --profile=- /srv/reproducible-results/rbuild-debian/tmp.wjmIrRuTSj/b1/golang-github-hashicorp-raft_1.1.2-1_armhf.changes /srv/reproducible-results/rbuild-debian/tmp.wjmIrRuTSj/b2/golang-github-hashicorp-raft_1.1.2-1_armhf.changes ## command (total time: 0.000s) 0.000s 1 call cmp (internal) ## has_same_content_as (total time: 0.000s) 0.000s 1 call abc.DotChangesFile ## main (total time: 0.258s) 0.258s 2 calls outputs 0.000s 1 call cleanup ## recognizes (total time: 0.017s) 0.017s 10 calls diffoscope.comparators.binary.FilesystemFile 0.000s 8 calls abc.DotChangesFile Wed Aug 25 09:31:41 UTC 2021 I: diffoscope 177 found no differences in the changes files, and a .buildinfo file also exists. Wed Aug 25 09:31:41 UTC 2021 I: golang-github-hashicorp-raft from bullseye built successfully and reproducibly on armhf. Wed Aug 25 09:31:42 UTC 2021 I: Submitting .buildinfo files to external archives: Wed Aug 25 09:31:42 UTC 2021 I: Submitting 12K b1/golang-github-hashicorp-raft_1.1.2-1_armhf.buildinfo.asc Wed Aug 25 09:31:43 UTC 2021 I: Submitting 12K b2/golang-github-hashicorp-raft_1.1.2-1_armhf.buildinfo.asc Wed Aug 25 09:31:44 UTC 2021 I: Done submitting .buildinfo files to http://buildinfo.debian.net/api/submit. Wed Aug 25 09:31:44 UTC 2021 I: Done submitting .buildinfo files. Wed Aug 25 09:31:44 UTC 2021 I: Removing signed golang-github-hashicorp-raft_1.1.2-1_armhf.buildinfo.asc files: removed './b1/golang-github-hashicorp-raft_1.1.2-1_armhf.buildinfo.asc' removed './b2/golang-github-hashicorp-raft_1.1.2-1_armhf.buildinfo.asc'